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ABSTRACT 

This report provides a synthesis of information 
currently available on the development of educational outcomes and a 
system of indicators for students with disabilities. An overview 
summarizes activities of the National Center on Educational Outcomes 
in the areas of model development , survey of state practice, 
information exchange, solutions to technical and implementation 
issues, and analysis of existing databases. The first section reviews 
educational reform in the United States and examines lessons from 
research reform, ideas from past reform, and the reform movement of 
the 1990s including implications for students with disabilities. The 
second section addresses current practice and definitions of outcomes 
and indicators as well as six current models of educational 
indicators. The third section identifies critical issues in 
developing a comprehensive system of outcome indicators related to: 
conceptual development; methodology; data analysis, interpretation, 
and policy making; and monitoring and changing the system. Considered 
next is the current status of outcome indicator activities in policy 
groups including general education outcomes activities (and 
implications for students with disabilities) and special education. 
The current status of outcome indicators activities is then 
summarized for general education (five states) and special education 
(11 states)* The conclusion is followed by a listing of approximately 
100 references. (DB) 
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Overview 



In October, 1990, the National Center on Educational Outcomes (NCEO) 
for Students with Disabilities was established at the University of Minnesota. The 
Center is a collaborative effort of the University of Minnesota, the National 
Association of State Directors of Special Education (NASDSE) and Sl Cloud State 
University. Its mission is to provide national leadership in the development of 
educational outcomes and a system of indicators for students with disabilities. The 
Center works with national policy-making groups, state departments of education, 
and other groups as it seeks to promote national discussion of educational goals and 
indicators of educational outcomes that are inclusive of students with disabilities. 
Through its work, the Center is helping to meet the following needs: 

Accountability: There is a need for a comparable accounting system across 
states on variables other than child count data. Special educators are 
increasingly held accountable for producing outcomes rather than just 
providing access to services. Outcome indicators are needed to assess this 
level of accountability. 

Program Improvement: Development of outcomes and indicators will 
provide the dependent variables that are needed in empirical 
investigations to assess effectiveness of interventions. Outcome indicators 
provide the data base or which sound conclusions and decisions can be 
made to guide the future of services for infants, children, and youth with 
disabilities. 

Policy Analysis: A common system of outcome indicators is needed for 
policy development and analysis to ensure that services and supports are 
consistent with public policy goals and are conducted in an efficient, 
appropriate and effective manner. 

Public Information: Congress, managers at ail levels, parents, and students 
themselves have a right and need to know the extent to which students are 
profiting from their educational experiences. Outcome indicators provide 
the data for public information. 

The National Center o?i Educational Outcomes has identified work plans for 
several activities to advance the development of systems related to the assessment of 
educational outcomes for individuals with disabilities. The primary activities of the 
Center arc structured to involve the participation of a network of key stakeholder 
groups, including parents, professionals, local, state, and national government 
representatives, persons from diverse racial, ethnic, and linguistic backgrounds, 
individuals with disabilities, researchers, and others. The major activities of the 
center include the following: 

Mod?l Development 

NCEO is responsible for developing a conceptual model of educational outcomes. 
For this model, NCEO will clarify the meaning of key terms, such as "outcomes," 
"indicators," and "system of indicators," and will conceptualize the outcome domains 
(c,g., achievement, participation, attitudes) that are believed to be important and that 
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provide information about the results of education. NCEO leads a consensus process in 
which stakeholders participate in refining the model and achieving broad 
agreement on outcome domains. Stakeholders involvement is also critical to the 
development of a system of outcome indicators, which are the specific kinds of data 
used to represent outcomes. A list of outcome indicators and the possible ways they 
interact with each other will be developed* Working papers on these topics will be 
subjected to continuous revision as new information is obtained from the literature 
on educational models* from stakeholders, from experts, and from data analysts 
activities. 

Survey of State Practice 

Another major activity of the National Center on Educational Outcomes is 
surveying states on an annual basis. Annual descriptions of state approaches to 
outcomes assessment in special education and general education will be prepared to 
inform State Directors of Special Education about existing practice. These 
descriptions will also help the Center to adjust the system of indicators to better meet 
the needs of states. The survey will also help to identify technical and 
implementation issues surrounding the assessment of educational outcomes of 
infants, children, and youth with disabilities. 

Information Exchange 

Information exchange is an essential activity of the Center because the 
outcome model and the indicators system are being developed through an interactive 
process in which key stakeholders have an important role. The annual report of 
state practices in outcomes assessment will be a primary part of information 
exchange. In addition, the Center is developing news briefs to address issues related 
to educational indicators, a set of technical issues bulletins to identify and address 
relevant technical issues related to outcomes assessment and data management, and 
an information synthesis series to provide summaries of information related to 
outcomes that can be found in existing literature or data bases. 

Solutions to Technical and Implementation Issues 

The Center also is serving as a vehicle for identifying technical and 
implementation issues and developing solutions to them. These issues are being 
identified through the state survey as well as through conversations with state 
directors and others. In addition to Center personnel, key resources will be 
consultants from a variety of fields, serving as a broad network of individuals who 
regularly address technical and implementation issues. 

Analysis of Existing Data Bases 

The Center will use existing data to answer current questions and to link state 
data with other existing data. Among the existing data sets arc those developed in the 
National Longitudinal Transition Study and in several states as part of federally 
funded evaluation grants. Major national data bases, including the National 
Assessment of Educational Progress, High School and Beyond, and others, arc being 
examined for information about individuals with disabilities. The Center is working 
with current data collection efforts to obtain better information on students with 
disabilities who are participating in assessments and students with disabilities who 
are excluded from participation. 



.The amount of information on educational outcomes and indicators is massive, 
and multiplying daily. Much of it is fugitive in nature; it is not in published articles 
and books so much as in drafts of documents being considered for policy making 
purposes. The synthesis of this kind of literature is considered to be a critical step in 
addressing issues related to assessing educational outcomes and in working toward 
the development of a comprehensive system of indicators. Shavelson, McDonnell, 
and Oakes (1989) have noted in their initial formulation of a model of education and 
potential indicators for RAND that the model was based on an "extensive review of the 
social indicator and educational research literature" (p. 9). For the National Center 
on Educational Outcomes for Studuits with Disabilities, it is necessary to give 
attention not only to the most m ent research related to outcomes indicators in 
general, but also to research that has been conducted in special education. Topics of 
relevance to this literature review include, at a minimum: 

• Educational reform and its current status in the United States 

• Definitions of key terms. 

• Current Models of educational indicators in both general education and 
special education 

• Current Status of Outcomes Indicators Activities in both general 
education and special education, including those of key national policy or 
policy-related groups (such as CCS SO, NGA, NCES), regional groups (such as 
SREB and Mid-South Regional Resource Center), and State Departments of 
Education 

• Critical Issues in the development of a comprehensive system of 
educational indicators 

The purpose of this document is to provide a synthesis of information that is 
available in the current literature. This document will be revised annually in order 
to incorporate new literature and current events that pertain to the educational 
outcomes of students with disabilities. 
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As we entered the 1990s, commissions, task forces, and public opinion polls had 
highlighted the presence of significant problems in U. S, educational practice. The 
lives of many children were viewed as being in disorder* This disarray was reflected 
in a slacking of scholastic effort, a decline in achievement, decreasing school 
attendance, and increases in crime, precocious sexual behavior, drug use, and 
alcoholism. For several years, reports had indicated that America's children and 
youth did not measure up in mathematics (NAEP, 1988a) v that the recoveries made in 
science achievement did not match previous declines (NAEP, 1988b), and that while 
some groups of students made gains in civics proficiency across a 12-year period, 
most did not (NAEP, 1990). Reports also indicated that America itself was at risk 
because it no longer had a competitive workforce (Mishel & Frankcl, 1990), one that 
was trained in the skills needed for success in a competitive international 
marketplace* As these findings became generally known in society and specifically 
known among state education agencies (SEAs), government agencies, parents, and 
members of Congress, there has been a press toward increasing accountability for 
educational systems. This turmoil in education and the resulting cries for 
accountability have ail become part of a massive reform movement. 

In order to understand the current reform movement, it is helpful to examine 
the concepts of "reform* and "change" in an educational context, and to review 
knowledge accumulated from past reform efforts and from research on these 
reforms. This background helps to clarify how the ideas, concepts, and values that 
dominate the current reform have developed. 

The notion of "reform" is as eld as the educational system itself and embraces 
the idea of a need for change. The educational system operates within a larger 
American society, one that changes continuously in order to achieve progress. 
American education is expected to prepare citizens to adapt Jo changing reality and to 
contribute to it (by maintaining change). In American society there seems to be a 
constant pressure to examine educational goals, directions, policy, and structure. 

In this section, the present reform movement is examined in terms of (a) 
lessons from research on educational reforms* and (b) lessons from past reforms. 
After these, the current educational reform is examined in light of this knowledge, 
with consideration given to the implications of the . reform movement for students 
with disabilities. 

Research on Educational Reform 

Extensive research on educational reform has emerged in the past several 
decades. While research efforts have focused on numerous issues (e.g., 
organizational change, policy and evaluation, Iocal«statc-n3tional initiatives, 
curriculum and instruction reforms, etc.), the studies can be divided into those that 
investigate the content of the reforms, and those that explore the politics of 
structural and procedural changes (or mechanisms and processes of change). The 
" content " refers to that which is the focus of change. Different reforms have 
focused on different issues, such as agreement on goals (or objectives, outcomes, 
results), curriculum innovations (types of knowledge, skills, and values), and 
instructional changes (approaches to teaching, processing, and interacting). The 



politics refer to the design of mechanisms (structures, roles, authorities) and 
processes (procedures, methods, activities) used to achieve the content of the reform. 
Each reform addresses the amount and type of restructuring in the power relations 
(within the system and between the system and external forces) . and/or resources 
that are required from the educational community, the government, or the total 
society. 

Content. Research on the content of reform movements has led to the 
recognition of patterns of educational reform. Rhoades and Sunshine (1990), for 
example, argue that every reform "put forward a goal or goals and a corresponding 
set of proposed reform actions to achieve those goals" (p. 6*7). In their examination 
of the major educational reforms in the United States, they found that one or more of 
five questions was being addressed: "(1) Who will be taught? (2) What learning 
should occur? (3) How should learning occur? (4) How will one know what has 
been learned? and (5) How much educational funding is needed to bring about the 
desired learning and how is that funding to be raised?" (p. 7). For example, the 
progressive movement that followed John Dewey's approach was mainly concerned 
with "How should learning occur?" The current movement of outcome-based 
education, on the other hand, is concerned with "What learning should occur?" and 
"How will one know what has been learned?" 

The fundamental questions addressed by a reform movement are accompanied 
by endorsement of certain values about education. Wirt, Mitchell, and Marshall 
(1988) contend that there are four major values that serve as driving forces in 
educational reforms and policy making. The first is the value of " quality. " which 
refers to the expectation that the educational system will provide "the best" for its 
students, by searching for the best teacher and administrator training, the best 
leaching methods, the best curriculum and instructional resources, or the best 
evaluation of students' and professionals' performance. The second value is 
"efficiency." Proponents of this value believe that resources should be used to gain 
the maximum possible. Efficiency is also the expectation that people in different 
parts of the system will be held accountable for their responsibilities. The third 
value is "e quity. " This refers to the belief that when a gap is found between the 
norms of social life and the needs of a certain group, public resources should be 
allocated to close this gap; more resources should be channeled to a disadvantaged 
group. Equity is the belief that every individual in a society has the same worth, and 
that society is responsible for recognizing this worth. The fourth value is " choice ." 
and it refers to the democratic belief that school clientele and educators should be 
able to exercise their sovereignty, to select preferred options from a range of 
available possibilities. 

While these four values all exist in American society, they reflect the presence 
of cultures that at times have had conflicting priorities. Each reform movement is 
characterized by endorsement of one or more of these values. The chosen values 
guide policy making and shape the reform movement. For example, a reform that 
calls for improving die quality of education by having standards of "excellence" or 
"proficiency" seems to be promoting the role of the professional elite (who set the 
standards), and to empower higher authorities to control the implementation of the 
standards. 

The fundamental questions, goals, and values endorsed by each reform 
movement are the basis for the decisions and actions that determine the focus of the 
reform. Regardless of the content of the reform, however, it needs to be realized 




through a thoughtful change process. The mechanisms and processes that enable 
reforms to take root is the focus of the second type of educational research. 



Mechanisms and processes of reform. ' Research on the politics of the structure 
and procedures necessary for educational change draws from the broader field of 
organizational change. It is generally recognized in the literature that regardless of 
the choser paradigm for educational reform, reform must progress through three 
broad stages (Berman et al., 1974): 

(1) Initiation. A need for change is recognized, and a decision is made to 
initiate a process in which information is gathered to understand the 
characteristics of the problem, the target population(s), the 
environment, the mechanisms and power-relations that need to be 
activated or changed, and the mode of operation best suited to the 
particular reform. 

(2) Implementation Specific strategies are formulated, the practical 
barriers and actions are identified, long-term as well as short-term 
working programs are developed, the linkage process between the groups 
that are involved is activated, and so is the diffusion (or dissemination) 
process. 

(3) Institutionalization. As the last stage, institutionalization is the process 
through which the new practices or products become familiar, and the 
initial difficulties get "worked out." 

While these three stages can be found in almost any educational reform 
movement, researchers (e.g.. Wolf, 1984; Scheurich & Iraber, 1990) recognize three 
dominant paradigms of change: the rational (or functional) approach, the cultural 
approach, and the political (or critical-theory) approach. 

Reforms that adopt the rational approach proceed by emphasizing technical 
and mechanistic orientations. The assumption behind this approach is that 
professionals who have the knowledge (e.g., research- based information), ability, 
and power can introduce change, usually in a top-down fashion. The change is 
introduced via two mechanisms: (1) Force: manipulation of rewards and 
punishment, mandates, or inducements: (2) Relearning (massive education, 
inservice training) and internalization; or some combination of the two. Educational 
reforms that follow this paradigm consider the administrator (e.g., legislators, 
superintendents, educational boards) as the reform experts and, thus, the initiation 
stage is most likely to begin at the top. The likelihood that these reforms will be 
accepted depends on the administrator's ability to interpret the environment, to 
recognize the groups that influence implementation (e.g., parents, community 
groups, teachers), to have a strong vision and a good working plan that will meet or 
shape the inteiest of the dominant coalition (Firestone. 1989). 

Reforms that adopt the culturalist approach emphasize a more sociological, 
rather than logical, perspective. The major assumption is that a reform would fail 
unless there is an understanding of the school culture as experienced within the 
school and outside of it in the community (Scheurich & Imbcr, 1990). It is further 
believed that all affected constituencies should be part of the change effort in order 
to create self-interest in the participants that would in turn enhance their support of 
the reform. Accordingly, the "old" norms, attitudes, and acceptable behaviors can be 



recognized only if all the constituencies are involved in a dialogue or a linkage 
process. A systematic and continued sharing of altitudes and values by people who 
represent the school's subcultures, would increase their awareness of a variety of 
attitudes and would create an open climate for reform efforts. The major criticism 
that emerges from analysts of cultural reforms (Scheurich & Imbcr. 1990) points to 
the fact that participation alone does not give adequate voice to all constituencies. It 
was found that those representatives that have more knowledge, power, and 
resources (in the community) have the power to have more influence over other 
members. They have the necessary skills and knowledge to persuade others to 
support their own needs, to protect and benefit their constituencies. 

The political approach to educational reform can be best illustrated in the 

critical-theory approach. It is assumed that educational reform requires the 
understanding of the "power" that each constituent has, and the relationships 
between the "powers." Sirotnik and Oakes (1986) argue that "during the process of 
critical inquiry, participants . . . become conscious of how current ways of schooling 
are grounded in the larger historical and social context of the culture as well as in 
the particular institutional and social context of the culture of the school" (pp. 36- 
. 37). The main problems associated with these reform efforts derive from the fact that 
not. all constituencies want to participate in all decision making, and not all of them 
are capable of arriving at the type of sociological understanding that is the goal of 
the process. Teachers, for example, generally are not interested in replacing the 
principal or the superintendent. They would like their voice to be heard and to be 
considered, but they can rarely participate in a process that requires engagement in 
the "whole picture." Rather, teachers have a better understanding of their 
particular concerns (Sarason, 1990). Parents, minority representatives, and other 
community members often lack the verbal capabilities to persuade and influence 
opinions as do experts and people who have backgrounds of power (Scheurich & 
Imber, 1990). In this respect, both the culturalist and political reforms suffer from 
the weakness of having difficulty learning from all constituents about the social 
environment and school culture, and using the mutual understanding to create equal 
opportunities for influence. 

Lessons from Reform Research 

Many lessons emerge from research that analyzes and interprets educational 
reforms (Sarason, 1990). Five lessons can be identified in this research. 

Lc&SfiD L U is important to recognize that the role of schools is to educate all 

children and not to solve all societal problems. While hew social problems, such as 
the growing number of homeless children, is a challenge that needs to be seriously 
addressed by schools, it is by no means the responsibility of schools to solve the 
problem of i.omelessness. Educators can no longer assume that they can solve all 
problems. They should recognize the complexity of the issues and admit that social 
problems that originate in the larger society should be addressed by multiple social 
structures (e.g., health system, housing, social security). It is the role of the 
educational system to understand its position within society, to collaborate with other 
social systems and to focus its reform efforts on addressing educational issues 
(Sarason, 1990). 

Lesson 2. Just as reform initiatives cannot assume responsibility for problems 
originated in the larger society, they cannot neglect to understand the relationships 
between the educational system and the diverse community groups and institutions 



that clearly interact with the system. Sarason (1990) argues that many past reforms 
have not succeeded in accomplishing their goals because the people who 
OfChcsiraicd those reforms failed to understand the educational system as a system. 
Most reformers have Hole knowledge of the' structure and power relations that 
characterize the system and consequently lack the ability to form a holistic 
conception of the system they seek to change. Moreover, a failure to view things 
holisticslry brings miscalculations to the selection of actions and the extent of their 
impact ea the target population. Thus, educational reformers need to adopt "a stance 
that would permit them to understand the nature of the system in its complexity" 
(Sanson, 1994), p. 45). 

f n ff»«ap 3. While adopting a system approach is by no means an easy task, 
especially since our understanding of the system is far from comprehensive, a 
prerequisite for successful reform is the proper alteration of power relationships in 
the system (Sarason, 1990). Educational research emphasizes the necessary change 
in the structure and in the flow of information and authority within the newly 
established structure. While this change is not sufficient by itself to guarantee the 
overall success of a reform, it is nevertheless a precondition. Sarason (1990) reports 
that many past reforms that failed to accomplish their mission did not address the 
need to change the pattern of relationships. Most reforms had asserted that there 
was nothing wrong with the existing structure, but rather that it was the lack of 
better people (better teachers, administrators, etc.) that was making the system 
ineffective. However, a reallocation of power and a change in role definitions was 
actually needed in order to confront those problems that arose from inappropriate 
and Ineffective mechanisms and processes. 

It is also important to realize that a change at one level of the system (e.g., the 
legislative or other policy-making level) is not going to bring the desired outcome 
unless "the phenomenology of individuals and their institutional relationships" (p. 
101) (e.g., teachers' practices in the classroom) is addressed as well. Policy change 
will fail to bring a change in practice unless change is implemented at each level in 
the system in a way thai encourages the unlearning of traditional customs and values 
in favor of the new. 

JLesson 4. Hie fourth lesson from educational research addresses the need to 
understand that each reform has its own focus. Recognizing the specific goals and 
the values associated with them is a prerequisite. The chosen content of a reform is 
an outgrowth of these goals and values, and it should be recognized as only one 
alternative among many possibilities. Consideration of alternatives does not reflect a 
weakness la a policy. Rather, public discussion is valuable since the complexity of 
the system demands open-mindedness about all possible solutions. For example, 
educational reformers whose goal is to raise the quality of the learning experience 
need to consider all possible alternatives (e.g., changing the cortent of the curricula, 
changing instructional methods or resources) in order to choose the best alternative. 

Lesson 5. A final lesson is related to all other lessons since it emphasizes the 
importance of integrating all aspects: educational reform needs to be conceptualized 
within the context of the larger society and should be thought of in terms of a 
system's change. While addressing the specific concerns at a specific point of time 
in history, past problems and reform efforts should be considered in order to 
understand the development of these problems. The particular form of action should 
determine both the focus of the reform (i.e., goals, values, content) and the systemic 
alterations in the structure of power relationships and the roles of ihe participants at 



cvciy level of the system. A change in focus has to go beyond the external decision 
making, and reach the internal thinking and practices of all those involved in and 
affected by the change. 

Ideas from Past Reforms 

In order to better understand the concepts that underlie the present reform 
movement, it is worthwhile going back to past reforms to find the ideas that are 
relevant and might have influenced the development of current thinking. Four 
ideas that have emerged in the past 40 years can be viewed as antecedents of the 
present emphasis on outcomes in education (Schleisman & King, 1990). 

Objectives. In 1950, it was proposed that "objectives" could serve as identifiers 
of both the behavior and the area in life to which the behavior applies. Tyler (1949) 
called curriculum and instruction planners to develop: purpose, content, 
organization, and evaluation. He emphasized the importance of setting goals before 
developing anything else. While the idea of objectives persisted, the focus has 
changed from behavior objectives to ones that state the outcomes of learning. 
Behavioral objectives were found to be more suited to the learning of basic 
knowledge and skills. Higher order thinking skills could not be stated properly in 
terms of precise behaviors toward which teachers could aim, but could be evaluated 
as cognitive outcomes. 

Measurement. In the 1960s another relevant educational concept was put 
forth. The importance of "measurement" as a way to know how much students have 
learned was emphasized. The measurement of choice, at that time, was norm- 
referenced testing, which involved the comparison of every student's performance 
with the scores of other students on the same test. While useful for comparison 
purposes; the shortcomings of this measurement concept soon became apparent. 
There was a lack of information about students' actual understanding and 
misunderstanding, and therefore it was not possible to provide continuous feedback, 
to consider the individual's needs, to know which objectives had been accomplished 
and whether the instruction was good (Schleisman & King, 1990). The importance of 
comparative measurement, however, had remained as a way to learn about the results 
of the educational process. 

Mastery learning. A third educational concept that emerged later in the 1960s 
was the idea of "mastery learning." Basing his model on John Carroll's (1963) theory 
of school learning, Benjamin Bloom (1968) developed ihe link between aptitude (i.e., 
time needed to learn, or quality of instruction and differential amount of learning) 
and achievement. He argued that some students need more time to master a unit of 
learning. Accordingly, unless one masters the learning objective there is no reason 
to move to the next instructional unit. This idea of accommodating to individual 
differences has persisted and been expanded to include all individualized programs 
for all subject matter, and a variety of instructional me;hods and materials to serve 
the needs of diverse students. 

Accountability. Another key concept came from the realization in the 1970s 
that many students fail to learn in school and that this failure is connected 
frequently to an inability to succeed in the world. Parents and community members 
demanded "accountability" mechanisms to ensure that school would be responsible 
for achieving certain standards. In the 1970s and early 1980s, the primary form of 
accountability was mandatory assessment and evaluation procedures used by some 
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states. Today, the concept of accountability, while changing its focus to include other 
methods and procedures, remains a viable term. 

The concepts of objectives, measurement 9 mastery-learning, and 
accountability are the major educational ideas that have contributed to the 
emergence of the present reform approaches. The most recent reforms have grown 
out of a multitude of reform initiatives that were generated in the 1980s. These 
initiatives were in response to a flood of reports abort the condition of education, the 
reforms needed, and the educational visions for the future. 

Three Wwcs of Educational Reform in the 1980s 

The educational reform reports of the 1980s appeared in three waves (see Table 
1)* The first wave was a flood of reports published in 1983, in which the focus was on 
the dangers to the nation's growth and strength that might result from mediocre 
education. Many reports emphasized the importance of commitment to excellence by 
advocating higher standards and more of the same: more courses, more homework, 
longer school day and year, and more state and local responsibility. In response to A. 
Nation at Risk and other reports, Secretary of Education Terrell Bell released for the 
first time in 1983 the "wall chart," which provided a comparison of states on a variety 
of educational indicators. In September 1984, President Ronald Reagan presented 
four target national education goals to be reached by 1990: (1) to raise the high- 
school graduation rate to more than 90 percent, (2) to raise scores on college* 
admissions tests above the 196S average. (3) to make teachers' salaries competitive 
with entry-level business and engineering graduates' salaries, and (4) to stiffen 
high school graduation requirements. 

The second wave of educational reports focused on improving school 
organization and policy and the quality of the nation's teachers in order to produce 
better results. In 1986 the governors published Time for Results: The Governors' 
1991 Report on Education. They challenged educators to find better ways of assessing 
progress, indicating they were willing to increase funding if educators could show 
the public that education is producing better results. This report and others (e.g., 
Investing in Our Children. First Lessons), emphasized the need for early childhood 
education for children at risk, and indicated that more state and federal officials were 
willing to take responsibility for educational reform. 

The third wave of reform started in 1987 with a call to give "the highest 
priority to early and sustained intervention in the lives of disadvantaged children" 
(Committee for Economic Development, 1987). Several reports, such as The Forgotten 
Half: Non-College Youth in America (W.T. Grant Foundation, 1988) and Turning 
Points (Camegie Council on Adolescent Development, 1989), emphasized the 
importance of dealing with the dropout problem, poverty, and the changing 
demographic characteristics of the American family. It was argued that school 
effectiveness in preparing all children for the 2 1st century should involve the 
reconstruction of the educational system. In addition there was a caM to reach 
consensus about the national educational goals and the strategies to achieve these 
objectives. 

Goal-setting efforts led to the September, 1989 educational summit of President 
Bush and the nation's governors in Charlottesville, Virginia. The summit created for 
the first time an agreement among all the states about the national goals in 
education. The Final Report (January 31, 1990), upon which all the governors 
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Table i 

Educational Reform Reports of the 1980s 



First Wave - 1983 

The Paidcia Proposal; — An Educational Manifesto (Adier. 1982). 

High School: A RfipflQ on AjBgjflM ^"ndgrv VAnanitm (Boyer, 1983). 

Action for Excellence: A Ccmprfhrnsivc Plan to Improve Our Nation's Schools (Education Commission of 

the States, Task Force on Education for Economic Growth, 1983). 
A Place Called School: Promecta for the Future (Goodlad, 1983) 
National Assessment Findings and Education Policy Questions (NAEP, 1982). 
A Nation at Risk: The Imperative for Educational Reform (National Commission on Excellence in 

Education, 1983). 

Meeting the Need for Quality: Action in the South (Southern Regional Educational Board Task Force on 

Higher Education and the Schools, 1983). 
Academic Preparation for College: What Student* Meed to Know and Be Ahte to no (College Board, 

Educational Equality Project, 1983). 
Making the Grade (Twentieth Century Fund Task Force on Federal Elementary and Secondary 

Education, 1983) 

Second Wave - 1086 

First Lessons, Phi Delta Kappan (Bennet, 1986) 

Investing in our Children: Business and the Public Schools (Committee for Economic Development, 1985). 
Literacy: Profiles of America's Voting Adults (Kirsch & Jungeblut, 1986). 
The Reading Report Card (NAEP, 1985). 

Time for Results: The Oovcmors' 1991 ggport (National Governors' Association, 1986). 

Results in Education: 12SJL (National Governors' Association, 1988). 

What Works (U.S. Department of Education, 1986). 
. Youth Indicators 1988. (Office of Education Research and Improvement, 1988). 

Third Wave - 

Results of an Inventory of State Indicators and Data on Science and Mathematics Fdncation (Blank, 1987). 
Turning Points; — Preparing American Youth for the 21st Century (Carnegie Council on Adolescent 
Development, 1989). 

Children in Need: Investment Strategies for the Fdnrati onallv Disadvantaged (Committee for 

Economic Development, Research and Policy Committee, 1987). 
Text of Final Summi t Statement (Crossroads at Charlottesville, 1989). 
Charting Change in Infants. Families, and Services: A Cmide to Program Evaluation for 

Administrators and Practitioners (National Center for Clinical Infant Programs, 1987). 
Three Draft Outlines Qf the Final Report of the Special Study Panel on Educational Indicators 

(National Center for Education Statistics, 1990b). 
NAESAC Outcomes Subcommitt ee: Idea Paper (National Forum on Educational Statistics 

Committee, 1989). 

America's Shame. America's Hone: Twelve Million Yn nth at Risk (Smith, 1988b. 

Educational Achievement; Explanation s and Implications of Recent Trends (Congress of the United 

Slates, Congressional Budget Office, 1987). 
National Fdncation fioaU (The White House, 1990). 

The Forfot^n Half: N on-College Youth in America (William T. Grant Foundation Commission on Work. 
Family, and Citizenship, 1988). 
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agreed, included six national goals for education. The goals addressed school 
readiness, school completion, student achievement and citizenship, mathematics and 
science, adult literacy and lifelong learning, and safe, disciplined and drug-free 
schools. The report also showed recognition of the need for accurate and adequate 
assessment of progress for the national goals to be meaningful. Measurements must 
be reliable, valid, comparable, and constructive. The most recent efforts are focused 
on the ways to restructure the system, and the new standards (quality indicators) and 
assessment procedures that are most appropriate for influencing and assessing 
needed reform. 

Overall, the reform movement of the 1980s reflected a growing understanding 
and commitment to improve the condition of education. The important issues that 
were raised were concerned with the need to improve education for all children, 
especially for the growing number of disadvantaged and at risk children. It was 
recognized that in order to prepare youth for the future and to provide the nation 
with a well-equipped work force, education must invest in teaching new basics as 
well as higher order cognitive skills. New definitions of literacy, learning, and 
educational outcomes, as well as new assessment tools were viewed as needed to report 
these changes to the American people (Bennet & McLaughlin, 1988). Finally, the 
need to include all levels of government was recognized. The federal government 
has committed itself to addressing broader societal issues that affect children's school 
performance, such as the growing rate of child poverty, early childhood 
intervention, helping disadvantaged children and children with disabilities, 
assuring school readiness, and supporting research and development.. State and local 
governments have committed themselves to provide financial support and greater 
flexibility, while at the same time demanding more accountability. This federal-state 
partnership has created a commitment to restructure the system and improve 
assessment procedures. Educational indicators were seen as being the main source of 
data to assess this commitment. 

The ideas reflected in the three waves of educational reports had a variety of 
manifestations in practice. Fuhrman and Elmore (1990), in their analysis of state and 
local reforms, argued that during the 1980s there was an increase in the policy 
making and fiscal roles of states. Governors and legislators relied heavily on 
regulation and monitoring technologies "to enhance the state's oversight capacity" 
(p. 84). 

States increased their influence over traditional local matters such as 
graduation requirements, testing, and certification requirements. Viewing the 
reforms from the local perspective, rather than the state's perspective, however, 
indicates that local districts and schools did not experience an increase in state 
enforcement activities. The lack of state capacities to create mechanisms that would 
assure compliance, such as new staffing and funding for the new responsibilities, as 
well as many conflicting slate policies and the overall inability to have a holistic 
approach, created major obstacles ^in the implementation of the reform agendas. 

By the late 1980s, the impact of educational research had started to be evident 
in practice. Influential ideas from research included such notions as: (1) learning 
exposure time is an important factor in meeting the needs of diverse students' 
learning styles and abilities; (2) higher order thinking skills must be developed, not 
just academic basics and behaviors; (3) the teacher's role in the classroom needs to 
change to more of a directive and facilitalive force and less of an authoritarian 
expert; and (4) more decision making about curriculum, instruction, teachers' roles, 
and monitoring student development should take place at the school level. These 
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ideas had the potential to influence school management, classroom arrangements 
(more . cooperative learning in groups; more individualized learning plans), and the 
quality of testing and data collection at the state and national levels. The National 
Center for Education Statistics (NCES) expanded its data collection to include new 
indicators of progress and process, and explored the development of new methods of 
measurement (e.g., portfolios, performance tasks). 



The Reform Movement of the 1990s 

The current reform movement in education is characterized by the inclusion 
of both content and political/structural concepts. One concern that is related to the 
work of the National Center on Educational Outcomes is the movement toward 
outcome-based education (OBE). Four concepts underlie OBE: (a) outcomes, 
(b) criterion-referenced measurement, (c) mastery . learning, and 
(d) accountability. 

Outcomes- "Outcomes," as used in OBE. refer to the results of instruction, the 
"knowledge, concepts, skills, and attitudes" (Johnson. 1967). or the knowledge, 
techniques (processes, skills), values, and affects (Oagne. 1966) that are the products 
of education (in Schleisman & King, 1990). Oronlund (1970) suggested that outcomes 
include two classes of products: (a) mastery outcomes, which are the minimum 
capabilities of a course of learning, and (b) developmental outcomes, which are the 
complex objectives that can always be developed further (e.g., thinking strategies). 
More recently, Spady (1988) and the Minnesota Department of Education (1989) stated 
that OBE requires a change in the approach to curriculum development. Outcomes 
should be viewed as "learner goals" and these goals should provide directions for the 
instructional process and the subject domains. Subject domains should be viewed as 
means to an end rather than dictating learning objectives. In Minnesota, for 
example, the role of the state in OBE is to define outcomes in terms of the knowledge, 
skills, and attitudes that students should accomplish. It is the role of the school 
district to establish a set of objectives to guide the instructional process and the 
choice of subject domains. The school's role is to develop a curriculum that leads to 
desired changes in learners, as defined by the objectives and outcomes. 

Criterion-referenced measurement. Many believe that defining, outcomes for 
learners and considering individual differences in mastery of objectives require a 
Change from the traditional form of evaluation, norm-referenced (NR) 
measurement, to criterion-referenced (CR) measurement. Schleisman and King 
(1990) explain that CR measurement involves "comparing a student's performance to 
a criterion or specified level of achievement" (p. 13). The measurement provides a 
continuum on which students' test performances can vary from no proficiency to 
perfect performance. Along this continuum, a point that reflects acceptable 
performance can be selected to provide a criterion for proficiency. In outcome- 
based education, it is necessary to establish the criterion before curricula arc 
developed and instruction takes place (Spady, 1981). While norm-re fcrcnccd 
measures have utility for policy making, at the state and national levels their utility 
is viewed by many as limited at the district, school, and classroom levels. Criterion- 
referenced measurement is seen ?s important for reflecting "what the individual has 
done well and what remains to be mastered" (Schleisman & King, 1990, p. 15). 
Moreover, many would argue that CR measurement has a place in state and national 
assessments as well. 
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Some criterion-referenced measures are still in a state of development (e.g., 
performance-based testing, portfolio methods, etc.) and are not yet established as a 
preferred form of evaluation in the educational system. There is a growing 
understanding, however, that CR measures have the potential to be linked to 
outcomes, to provide data that can inform instruction, and to contribute to evaluation 
of practices and to decision making. 

Mastery learning. A concept that is related to criterion-referenced 
measurement is "mastery learning." OBE reflects the philosophy that "time spent" in 
class cannot serve as an indicator of mastery of content. It is believed that different 
individuals need different amounts of quality instruction and time to achieve the 
same objective. The idea that students should be allowed to experience learning 
where learning is "the constant and time becomes the variable" is consistent with 
OBE (Minnesota Department of Education, 1989). CR measures can provide educators 
with information about the mastery levels of individual students and about the 
additional learning experiences that are needed to achieve mastery of particular 
objectives. 

Accountability. Since the 1970s, educational accountability systems have been 
characterized by their accounting strategies and cost control methods, by their 
emulation of business models, and by their evaluation of education according to 
behavioral objectives. During the 1980s, a wave of state legislation began focusing 
on school report cards, merit schools, and interstate achievement comparisons (Kirst, 
1990). The concept of accountability emerged again as one of the primary concepts 
in the 1989 Educational Summit at Charlottesville, and again there have been attempts 
to clarify the confusion about educational accountability. 

Kirst (1990) recognized six approaches to accountability at the state level, 
noting that the type of accountability strategy most appropriate for a particular state 
is determined by the state and local context. He also noted, however, that a well- 
designed accountability system would probably include a sophisticated combination 
of several approaches. The six approaches, according to Kirst (1990), are: 

(1) Accountabil ity as performance reporting. It is assumed that the 
information in performance reports (e.g., state assessment, NAEP) can 

. stimulate activities to improve education. 

(2) Accountability through monitoring and compliance with standards or 
regulations. The most used techniques are regulations that aim at 
procedural compliance. Schools and districts that comply are regarded 
as accountable. 

(3) Accountability through incentiv e systems. Rewards are given to those 
who achieve desired results. Programs such as performance-based 
accreditation and teacher merit pay are examples of this type of 
accountability approach. 

(4) Accountability through reliance on the market . The public is 
empowered to choose- among schools, and "successful schools" arc 
getting more support as a consequence of market changes. The 
Minnesota open enrollment plan is an example of this approach. 
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(5) Accountability through changing thft focus of minority or cnntml of 
schools. It is assumed that schools will be more accountable if control 
over educational policy shifts from one group (e.g., superintendents) to 
another (e.g., parent advisory councils). School-site decentralization 
and community-controlled schools are examples of this approach. 

(6) Accountability through chanpinp professional roles. It is assumed that 
accountability can be achieved by providing professionals (e.g., 
teachers) with new roles or by creating mechanisms to assure 
professional standards. For example, having teachers review each 
other for tenure, or having teachers participate in site-based policy 
making are possible mechanisms. 

In an analysis of the accountability systems of several states, Kirst (1990) 
found that each system has its strengths and weaknesses. He believes that a 
successful accountability system would incorporate elements of all six approaches, in 
a manner that best fits the state's political and educational context. Although several 
states have attempted to employ many strategies, too often the policies have not been 
interrelated and at best have created a fragmented and incomplete accountability 
system. Many of the limitations of current state accountability systems are also 
related to a lack of explicit educational objectives or goals, a limited number of 
effective ways to influence schools, and major gaps in evaluation and monitoring 
methods. 

It has been argued that difficulties associated with the implementation of 
accountability policies are related to misconceptions about . control (Fuhrman & 
Elmore, 1990). Focusing on state control versus local control erroneously suggests 
that one bureaucracy has control over the other. This adversarial view, according to 
Fuhrman and Elmore (1990), stems from three confusions, "confusing activity with 
control, confusing control with influence, and generalizing too broadly about state 
influence on local discretion" (p. 89). In contrast to these confusions, the authors 
note that states that seem to generate many activities do not necessarily exercise 
more control over local districts. They also note that those states that have 
significant effects on local pgencies 

rely more on multiple mechanisms of influence than on direct control. 
Among the mechanisms of influence that states use are the mobilization of 
professional and public opinion and the use of information about performance 
to shape the policy environment within which local school districts operate, 
(p. 90) 

A state's influence is experienced differently by different districts, and this 
influence varies by policy area. Fuhrman and Elmore also identify three confusions 
about local control: "portraying local districts as units rather than active multilevel 
policy systems, viewing local discretion entirely as a function of the degree of 
higher level regulation, and overlooking the ways in which local actors influence 
that higher level" (p. 91). 

A more productive way to conceptualize state-local relationships is to focus on 
"state influence on the classroom, with school policy and district policy mediating or 
amplifying that influence" (p. 91). The "control" of districts over schools by means 
of inspection and enforcement systems is limited. However, districts do have the 
power to influence schools and to create real effects in classrooms, if they have 
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personnel with the knowledge, skills, and devotion to reform efforts. It is important 
to find the people at the district level who can work at the school site to mobilize 
change. As suggested by Firestone (1989), it is also important to: 

(1) Clarify the goals and provide the district with a vision. 

(2) Obtain resources, such as time, personnel, materials, facilities, training, 
and technical assistance, depending on the nature of the reform. 

(3) Provide encouragement and recognition. 

(4) Adapt "old 99 procedures (e.g., course sequences, textbooks, evaluation 
procedures) to new policies, 

(5 ) Monitor reform efforts by systematically examining and evaluating the 
reform. 

(6) Handle disturbances, from internal or external sources, that attempt to 
undermine implementation of reform. 

(7) Form district-school linkages to enhance the flow of communication 
and the exercise of influence. 

Accountability in education needs to be better understood. However, there is 
wide consensus that since all students can learn and deserve to learn, and that 
schools should achieve a certain criterion of performance, there is a need to create 
mechanisms to assure the accountability of schools. 

Outcome-based education (QBEV OBE emphasizes the importance of 
publishing "outcomes" as the knowledge, skills, and attitudes that all students need 
to internalize. Curricula should identify content areas and instruction should 
provide the experiences that can lead to these end products. To ensure that outcomes 
are achieved, it is necessary to create monitoring and evaluation systems. 

At the local level, OBE's use of criterion-referenced measurement and 
sophisticated accountability systems is believed to provide the mechanisms for its 
success. Development of outcome indicators is the first step in that direction. 
Agreement on the desired outcomes of education at both the national and state levels, 
and consensus about what should be regarded as outcome indicators can promote 
local understanding of the objectives of education, and can gride the improvement of 
our evaluation and accountability systems. 

Implications of the Reform Movement for Students With Disabilities 

Most reform initiatives of the past responded to declining academic 
achievement of a certain group in the student population and at times of the whole 
population. These initiatives represent an approach of "problem-solving" rather 
than one of constructive and preventive altitude. The emergence of OBE represents a 
shift toward the preventive approach. It is believed that if schools could find ways to 
meet the diverse needs of all students, many more educational issues would be 
addressed. While past initiatives attempted to achieve excellence by establishing 
more rigorous academic requirements, and increased testing for all students, recent 
efforts focus on the need to adjust the organization of services within schools and the 
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establishment of alternative paths to excellence. OBE calls for tbe creation of a 
system . in which the "outcomes" or objectives for all students are the same, but the 
routes and the pace for achievement vary with the circumstances of the individual 
student. Students with disabilities would be able to participate in the educational 
process of regular classrooms if special education services and expertise were used in 
a unified system. 

While special education still is a relatively separate system of services, where 
students with disabilities are "pulled out" from general classrooms in order to receive 
special services in separate locations (National Council on Disability, 1989). recent 
reform initiatives call for a change in attitudes toward disabilities, and 
organizational change to accommodate these attitudes. It is believed that the growing 
population of students with special needs is so substantial, that it becomes impossible 
to label as "different" and segregate so many students into a separate educational 
system. 

An effective approach toward enhancing the learning and achievement of 
children with diverse learning and personal characteristics requires a new 
pedagogy, where resources and responsibilities are shared by the general classroom 
teacher and special education specialists. Special educators can bring into the 
general classroom their unique understanding that 

no two students learn at the same rate or in the same manner.. ..the pedagogical 
implications of the effective schools literature suggest that one effective role 
for special educators is as consultative teachers, assisting regular classroom 
teachers in devising and delivering learning programs for individual students 
and consulting about instructional strategies and classroom aides. (National 
Council on Disability, 1989, p. 36) 

The reform movement calls for new teaching models that emphasize the 
importance of collaborative, small group work in which students help their peers, 
and the teacher consistently provides individualized feedback, direction, and 
instruction. These new teaching models require the breakup cf . the traditional 
classroom structure that is based on age or grade. A growing body of literature from 
the "Regular Education Initiative" (REI) provides new ways to understand effective 
partnerships, least restrictive environments, and more fully actualized 
mainstreaming. Jenkins, Pious, and Jewell (1990) argue that REI can be expressed in 
many ways according to local needs. However, they believe that several assumptions 
can guide reform at the classroom level. Regular classroom teachers are responsible, 
accordingly, for 

(1) educating all students assigned to them, ... (2) making and monitoring 

major instructional decisions for all the students in their 

class, ... (3) providing instruction that follows a normal developmental 

curriculum, ... (4) managing instruction for diverse populations. ...and 

(5) seeking, using, and coordinating assistance for students who require more 

intense services than those provided to their peers, (p. 481-483) 

REI implies that special services should be integrated as much as possible in 
the regular class environment. Only students who need services that are highly 
specialized (e.g., Braille instruction for the blind) would need special classes. 
Integration, according to REI, involves the restructuring of the school environment 
info a place where service delivery occurs in the regular classroom. Principals, 
administrators, and policy makers have the responsibility to prepare the structure, 
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the resources, and the knowledge (e.g., teacher preparation) to enable the reform to 
take place. 

The development of outcome indicators for monitoring the educational status 
of all students' needs to begin with an understanding of the definitions of educational 
indicators in general, and outcomes indicators in particular, and with clarification of 
the issues that need to be addressed as indicators are identified. In the next section, 
the following topics are discussed: (1) current practice and definitions of outcomes 
and indicators, (2) current models of outcomes indicators, and (3) critical issues in 
the development of outcomes models and a system of indicators. 
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Current Practice and Dpflnitlnns nf Outcome and Indicator* 

When reading through the reform literature, one is struck by the variety of 
ways in which the same terms are used. For the National Center on Educational 
Outcomes, it is important that clear definitions and assumptions be delineated before 
beginning the actual development of a conceptual model of educational outcomes for 
children and youth with disabilities. This need is being met through the 
development of a Working Paper that will be open to continual review and revision 
as the Center's activities proceed. Included here is a review of the ways in which 
terms are used and defined in the literature and the initial Working Paper of the 
National Center on Educational Outcomes. 

Before proposing working definitions of ihc key terms "outcomes" and 
"indicators," it is helpful to examine the ways in which the terms are currently 
applied in practice. Part of the reason for doing this is that it highlights the variety 
of interpretations and the various levels at which the terms have been applied. It is 
also important to define these terms so that the same meaning is used by many 
people, especially if consensus is to be reached about a comprehensive system of 
educational indicators. Although various uses of the terms "outcomes" and 
"indicators" by states are included here, a more comprehensive review of state 
activities is included in a later section of this document. 

Current Practice 

Identifying outcomes. With all the interest in outcomes, many organizations 
and groups have identified what they believe should be the outcomes of education. 
And, of course, they talk about outcomes at differing levels: for individuals, districts, 
states, and the nation. 

Many attribute initial interest in outcomes to the early work of Ralph Tyler 
(1949) who, in his textbook Basic Principles of Curriculum and Instruction, called for 
specification of behavioral objectives. Early work on outcomes was enhanced by the 
work of Benjamin Bloom, who published a taxonomy of educational objectives, and by 
Robert Mager (1962), who prepared a seminal work on preparation of behavioral 
objectives. This early work was about the specification of behavioral objectives to 
guide the instruction of individual pupils. Mauritz Johnson (1967) contributed to the 
knowledge base on outcome-based education when he specified a curriculum that 
consisted of a structured series of intended learning outcomes. He talked about 
outcomes as products including knowledge, concepts, skills, and attitudes. Gronlund 
(1970) differentiated two kinds of outcomes: mastery and developmental. Others 
referred to these two types as "terminal" and "enabling." 

The lists of outcomes that individual researchers and writers have specified 
are primarily at the level of the individual. For example, Gagne (1974) listed five 
major categories of learning outcomes: verbal information, intellectual skills, 
cognitive strategies, attitudes, and motor skills. Over lime, the lists have expanded to 
include outcomes at the school district and slate levels. 

Several of the lists that have been generated by or for various policy groups 
arc summarized in Table 2. Because states also have an interest in identifying the 
outcomes of their educational programs, many have been formulating lists of their 
own. Some of these are shown in Table 3, which includes the outcomes listed by 
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Table 2 

fifrflmP >QS Qf PQ' 1 ^ Groups' Outcomes Lists 



Source Outcomes 



General Education 



Chicago Board of Education 
(Mrst, 1990) 



Council of Chief Slate School 
Officers (SeWen, 1990) 



Department of Labor, 
Employment, and Training 
Administration (1990) 



National Education Statistics 
Agenda Committee, NCES 
(NHSAC, 1990) 



Oakes (1986) 



Policy Studies Associates 
(McColtum & Turnbuil, 1989) 



Seiger-Ehrenberg (1985) 



Graduation 

Reading Achievement 
Mathematics Achievement 
Attendance/Dropout Rate 
Grade Retention Rates 

Attendance 
Achievement 
Attainment 
Affective Status 
Post-School Status 

Learning to Learn 

Reading/Writing/Computation 

Communication 

Creative Thinking/Problem Solving 
Seif-Esteem/Goai Setting 
Motivation/Personal and Career Development 
Interpersonal Skills/NegotiatlofVTeamwork 
Organizational Effectiveness/Leadership 

Achievement (core content and other subjects) 
Participation and Progression (dropouts, completes) 
Status after High School (postsecondary enrollment, 
entry Into labor force, employment. 

Achievement 
Participation 
Dropout Rate 
Attitudes 
Aspirations 

Achievement (subject knowledge, basic and higher order 

thinking skills) 
Attainment (graduation, GED, dropout rates) 
Postsecondary Experiences (employment, college 

attendance) 
Satisfaction/Engagement in School 
Citizenship 

intelligent and ethical action to accomplish tasks that 
society legitimately expects of all its members 

Establishes and pursues worthwhile goals of own 
choosing 
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Table. 2 (continued) 



Source 



Outcomes 



ShaveJson, McDonnell, & Oakes 
(1989) 



Special Study Panel on 
Educational Indicators (Indicators 
Panel, 1989) 



Stoddard (1961) (cited by 
Reynolds & Lakin, 1989) 



Special Education 

Mid-South Regional Resource 
Center (1986) 



National Council on Disability 
(1990) 



V 



SRI National Longitudinal Study 
(1989) 



Achievement (knowledge, understanding, and use of 

concepts, skiQs in mathematics and science) 
Participation (wfthin and outside school) 
Attitudes and Aspirations (including self -confidence) 

Achievement 
Attainment 

Postsecondary Experiences 
Beyond School Experiences 

"CufluraJ Im pe ratives': 

Ue Maintenance Skills 

Basic Literacy SkiUs 

Social Skais 
• Orientation toward Self-Development 



Student Performance 
Competencies 

Behaviors and Attitudes (attendance, suspension, 

dropout rates, and completion rates . . . ) 
Satisfaction 

Post-School Status (postsecondary schooling, 
employment, income, integration) 

Academic Achievement (course standards, graduation 

rates, number going to college, degree attained) 
Work Readiness (Job standard aspirations, number 

gainfully employed, number unemployed) 
Quafity of Life 

Internal (job satisfaction, coping skills, self-esteem) 
• External (sufficient income, clean and sanitary 

housing, involvement in community activities) 

Education 
Employment 
Independent Living 
Postsecondary and Adult Services 
Postsecondary Status 
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Tfitifs 3 

Examples of States' Outcomes Lists 



Stat* 



Outcomes 



Connecticut 



Florida 



Iowa 



Nebraska 



,*ow York 



Participation, attendance, graduation rates, suspension 
Academic achievement in math, writing, and reading on 

Connecticut Mastery Test 
Attitudes and attributes: motivation, persistence, 

satisfaction 
Graduate foflow-up 

f Mathematics 

Communication: reading, writing 
Functional literacy 

Background information 

High school program: course type, work experience, 

extracurricular activities 
Evaluation of school experience from student perspective 
Current lifestyle: marital status, Irving arrangements, 

leisure activities 
Post and current employment 
Contact with adult service providers 

Employment history 
Independent Irving functioning 
Financial assistance 
Communication skills 
Earnings 
Job skills 
Social skills 

Attendance, promotion, graduations and dropout rates 
Achievement on State and Regents exams 
District-level standards 



New Hampshire 



Oregon 



School based measures of: absenteeism, suspension 
rates, dropout rates, graduation rates, school 
satisfaction 

Residential setting, independent living situation, 
recreational activities, absence of social 
maladjustment, social relationships, personal 
satisfaction, employment 

Program and IEP description 

Parent and teacher survey on student's satisfaction and 
quality 
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several states that participated in a discussion held in Spring, 1990 by NASDS£L The 
activities of ail of these states, and others as well, are summarized in the last section 
of this document 

The variability in the lists in Tables 2 and 3 reflects at least three major 
influences on the kinds of outcomes included in a list First, while many people and 
groups want information about the outcomes of educational programs, they want it 
for different reasons. Some want the information for the sake of building a 
knowledge base. Others want outcome information for specific purposes already 
determined to be of importance, such as identifying outcomes needed for successful 
workforce involvement, or outcomes needed to attend college, or outcomes needed for 
a happy adult life. 

Second, the level of aggregation at which outcomes information is desired is of 
importance to different groups. Currently, information aggregation ranges from 
specific individual student data to averages based on national samples of students* 
Outcomes data may be desired for evaluation of programs at the local school by 
district level officials. Outcomes data may be desired at the state level to make 
decisions about funding districts (either to boost funding for poorly performing 
districts or to withhold funding from districts not performing up to expectations). At 
the national level, data may be desired to make decisions about where emphasis 
should be placed in funding (e.g., in science education to spur more career choices in 
this area). Data may also be desired at the international level to make comparisons 
among countries. Many argue that the specific outcomes that are identified should 
be different at different levels. 

A third influence on the content of a group's or state's list of outcomes is 
background student characteristics. In general education, some have identified 
different outcomes for college-bound students and vocational program students. 
Within special education, degree of severity of disabling condition is thought by 
some to be a key determinant of the outcomes information to be collected. Another 
common consideration is developmental or educational level* Outcomes identified for 
preschool education are said to be different from those identified for secondary 
education. 

The influences of why, at what level, and for which students outcomes data are 
wanted are evident in the lists provided in Tables 2 and 3. In Table 2, for instance, 
Seiger-Ehrenbcrg (1985) and Stoddard (1961; cited in Reynolds and Lakin, 1989), 
provide examples of a very broad view of desired student outcomes, one that is more 
philosophical in nature* By contrast, the practice of policy makers or policy- 
oriented groups (e.g., Kirst, 1990; McCollum & Tunibui!, 19S9; Scldcn, 1990) often is to 
list more specific outcomes in areas such as achievement* participation, dropout rate, 
and attitudes and aspirations. Practitioners sometimes include slightly different 
outcomes in their lists, choosing to include even more specific outcomes than do 
policy groups. 

As is evident in Table 2 f outcomes lists also reflect the level at which the group 
generating the list focuses its efforts. For example, one breakdown of level is the 
local, state, national, or international aggregation of data* Another breakdown is by 
age, level of schooling, or life stage, For example, the National Council on Disability 
and the United States Department of Labor arc primarily concerned with outcomes 
related to adult, work-related issues. Some states (e.g., Connecticut) address outcomes 
as a function of age or grade level in school, as well as post-school outcomes. 
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Another influence on the identification of outcomes in practice is the 
orientation of the development group toward the assessment of outcomes. For 
example. Michigan's orientation is toward developing extensive outcomes lists for 
each categorically-defined disability. The lists are highly organized around tasks 
and objective-referenced criteria, thus lending themselves easily to traditional 
assessment methods. Connecticut, by contrast, has a list of outcomes that reflects, in 
part, the state-wide mastery test As a result of negative reactions to traditional tests, 
Connecticut is focusing considerable effort on alternative/performance-based 
assessments of outcomes. 

Identifying indicators for outcomes. Although there has been much 
discussion on outcomes in the literature, and lists of outcomes are relatively easy to 
find, considerably less formal discussion of outcome indicators can be found. The 
dearth of information is especially noticeable in the application of "indicators" in 
practice. Examples do exist, however. The most well-known are the indicators used 
in the Wall Chart by the U.S. Department of Education to show educational outcomes. 
The indicators of education used in the Chart are SAT and ACT scores. The National 
Assessment of Educational Progress (NAEP) and the Assessment Program of Alberta 
Canada provide other examples. 

Indicators of outcomes in the National -Assessment of Educational Progress are 
scores on its tests. NAEP now is developing standards against which these scores can 
be compared. Most recently, NAEP published national standards for student 
achievement in mathematics in grades 4, 8, and 12.. Students' actual performance 
will be compared against standards for basic, proficient, and advanced achievement, 
thus providing an indicator of the educational achievement of the students. 

The Council of Chief State School Officers (CCS SO) has identified outcome 
indicators in the areas of science and math. Similar to indicators proposed for NAEP, 
the CCSSO indicators do not reflect any recognition of students with disabilities or any 
distinction between general and special education. 

In Alberta Canada, outcome indicators have been identified through an 
initiative called the Education Quality Indicator Project (EQI). The EQI sampled 20 
school jurisdictions regarding the indicators reported by the schools. The 16 
indicators it identified to assess student outcomes are shown in Tabie 4. Alberta also 
differentiated indicators used in general education and special education (primarily 
focusing on students with learning disabilities). Also identified were indicators 
currently applied in p r actice versus those considered to be desirable indicators of 
student outcomes- The types of indicators for general and special education were 
very similar, but the importance assigned to the indicators varied between actual 
practice and desirable status (e.g., employment after schooling becomes very 
important on the desirable list). 

Definitions 

Qutcorncs. An outcome is any product of the interaction between the 
individual and school experiences,- which in turn are influenced by the individual's 
life experiences. Individuals, including those with disabilities, differ on a number of 
individual characteristics and competencies, including their skills, abilities, 
temperament, and motivation to learn. These individuals attend schools and engage 
in life experiences where they learn and acquire skills. The current preoccupation 
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Table 4 

Alberta List of Outcome Indi, 



Teacher and Other Professional Assessments (tests, report cards, 

anecdotal data) 
Alberta Achievement Test Program 
Alberta Diploma Examinations 
Standardized Tests (Formal and Informal) 
Parent/Grade 12 Student Satisfaction Survey 
District Standardized Achievement Tests/System-Wide Exams 

(criterion referenced) 
School Evaluations/Reviews 
Drop-Out Rates 
Graduation Rates 
Teacher Observations 
Participation in Co-Curricuiar Activities 
Individualized Program Plans (IPP) 
Self-Esteem Measures 
Thinking Skill Measures 
Attendance Records 
Number of Rutherford Scholarships 
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with results reflects "the new definition, now struggling to be born" of education, 
one in which "education is the result achieved, the learning that takes root when the 
process has been effective" (Finn, 1990, p. 586). Hie particular outcomes one focuses 
on depend on one's perspective regarding education and educational outcomes* For 
young children with disabilities, educational services are mandated beginning at age 
3, and in some states services are provided from birth* Also, educational services are 
to be provided beyond the typical graduation age of 18 years, to encompass up to age 
21 years or higher in some states. This means that relevant educational experiences 
are not limited to grades K-12. 

As it begins to work with key stakeholders in developing a model of outcomes 
for students with disabilities, the National Center on Educational Outcomes (NCEO) is 
adopting the following definition of "outcome"; The result of interactions between 
individuals and educational experiences. The term "educational experiences" refers 
to learning experiences that are planned, managed, and organized by schools and 
other agencies. The "individuals" who are being served are infants, children, youth, 
and adults. The "results" are the effects that are brought about by the "interactions" 
between the individuals (and their particular circumstances and experiential 
background) and the educational experiences that are planned by the schools. 

NCEO also suggests that outcomes should have the following characteristics; 

• Be valued by society; 

• Reflect the degree to which essential cultural expectations have been 
attained in such areas as participation and achievement in normal 
society, education, training, employment, leisure, and community 
environments; 

• Include both direct and indirect results of educational experiences; 

• Be relevant to ail individuals* regardless of personal characteristics; and 

Be a product of learning and experience (what has been learned) rather 
than how learning occurred. 

Indicators . An indicator is a number, index, feature, or measure that allows 
for comparisons to be made. The general public thinks of economic indicators like 
the Consumer Price Index, the rate of unemployment, and the Gross National Product. 
Educators talk about input, process, context, and outcome indicators. Educators and 
evaiuators often use such terms interchangeably, and sometimes talk about 
"outcomes indicators," and at other times talk about "indicators of educational 
outcomes.* These different perceptions and uses of terms often cause confusion. 

Many organizations and agencies have developed indicators to measure the 
health of our educational system. For example, the National Center on Education 
Statistics has developed outcomes indicators in reading, math, and social studies; the 
Council of Chief State School Officers has developed indicators in science and math. 
Different indicator systems often have different purposes, audiences, and types of 
indicators. While it is clear that educational indicators are needed, there is little 
consensus on their definition, their uses, or on the types of indicators that should be 
employed. 
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Smith ; 1988a) r\iggests that "most commonly, an educational indicator either 
assesses or is related to a desired outcome of the educational system or describes a 
cons feature of that system" (p. 487); He does not suggest that the same indicators can 
be used at these different levels. An indicator generally is expressed as a number 
and is used to enable comparisons over time or among institutions, states, or nations. 
Smith emphasizes the importance of indicators being feasible to gather, valid, and 
useful for making policy decisions at some specific level, such as national, state, 
local, school, or classroom. 

While an indicator is generally expressed as a number, not all indicators are 
quantitative in nature. Shavelson, McDonnell, and Oakes (1989) state that some of the 
properties of educational systems are not directly or perfectly measurable. They use 
as an example the notion of "teacher quality." An indicator of teacher quality might 
include the aggregation of several statistics (like years of experience), assumptions 
about what those statistics imply (teachers with more experience are "better"), and 
qualitative judgments. Indicators are statistics, but not all statistics are indicators. 

Selden (1990) describes steps in developing working concepts about the notion 
of indicators. He states that: 

The first notion is that an indicator is a piece of data about education that is 
useful for planning or policy making. Indicators are not seen as some special 
class of information ~- higher order indices or complex composites. They can 
be relatively ordinary information, such as school enrollment data, that 
support planning or allow policy questions to be answered. Analyzed over 
time, used to compare units, or examined in relation to a social goal or 
standard, indicators are statistics that reveal trends, show where relative 
strengths and weaknesses lie in a system, and report how we are doing in 
relation to how we want to do. (p. 384) 

Policy makers speak of at least four different types or categories of indicators: 
input, context, process, and outcome indicators. These types of indicators are 
described and differentiated below. 

In piit Ipjd icat flEL These refer to the financial and human resources that are 
available to the school for use in meeting its mission and objectives. Inputs 
include fiscal resources, teacher quality, and the specific kinds of students 
who arc served. 

Context Indicators . Schooling is completed in a context, and students gain life 
experiences in specific contexts. Context indicators include indices of the 
characteristics of schools, and characteristics of the communities in which 
students live and develop, Socioeconomic status (SES) is among the most 
frequently identified context indicators. 

Process Indicators . Indices of process include "the adequacy of the curriculum 
and instruction received by students, the nature of the school as an 
organization in pursuit of educational excellence and equity." (Shavelson 
et ah, p. 7) These include the kinds of things identified in effective schools 
research (Walbcrg, 1984), as characteristics of the instructional 
environment (Ysseidyke & Christenson, 1987), and as critical instructional 
factors (Christenson, Thurlow, & Ysseidyke, 1989). 




Outcome indicators. Outcome indicators are indices of the products of a 
reciprocal interaction between the individual and school or life 
experiences. They can include student-level variables such as academic 
achievement, attitudes and aspirations, life adjustment, and post-secondary 
status (employment, education), and institution-level variables such as 
graduation and dropout rates. 

NCEO defines "indicator" in its working-paper as: a symbolic representation of 
one or more outcomes (or inputs, contexts, or processes) that can be used in making 
comparisons. NCEO focuses its activities on outcome indicators, rather than input, 
process and context indicators. The term "symbolic representation" is used in the 
definition instead of the term "number" because indicators can be quantitative 
(numerical) and qualitative. Indicators should have the following characteristics: 

• Reflect the desirable outcomes of education (e.g., achievement, 
attitudes); 

• Reflect particularly significant aspects of schooling or areas of policy 
interest; 

• Include both individual-level and system-level representations of 
outcomes; 

• Be based on procedures that are concise, timely, reliable, and valid; and 

• Use procedures that are already available, or feasible to derive from 
existing data, if possible. 
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Current MnrfPl* nf EdiirnHnniil Indicators j 

5 

NCEO believes that It is necessary for a comprehensive system of outcomes * 
indicators to be based on a sound conceptual model. This model may require 
specification of not only the domains of assessment, but also of the developmental 
levels at which specific outcomes are expected. The outcomes model also may need to 
indicate the extent to which indicators and their assessment should differ as a 
function of type of disability or severity of disability. 

Furthermore, the conceptual model of outcomes should be linked to a 
comprehensive model of educational indicators. This point has been made by others 
who argue against focusing on outcome indicators in isolation. Oakes (1989) has 
asserted that valid and useful indicator systems will include assessments of school 
context (inputs and processes) as well as outcomes. She says this broader focus will 
keep educators from identifying only narrow outcomes that will help them look good. 

The purpose in this section is to review several broad conceptual models that 
have been proposed by others to show linkages among educational inputs, contexts, 
processes, and outcomes, or some subset of these. Most of these models have not been 
very detailed, but rather have presented listings of outcomes. It is important, 
nevertheless, to review these models before attempting to develop a conceptual model 
of outcomes for cedents with disabilities. 

Carroll's Model of School Learning 

Carroll (1963) was one of the first to specify determinants of student outcomes. 
Carroll indicated that outcomes are an inverse function of the relationship between 
time spent learning and time needed to learn. He viewed time spent learning as 
consisting both of perseverance and opportunity to learn. Time needed to learn was 
viewed by Carroll as differing for individual students and as the product of student 
aptitude, ability to understand instruction, and the quality of instruction. Carroll's 
model of the factors determining instructional outcomes (school learning) is shown 
in Figure 1. 

O akes' Model of the Educational System 

Oakes (1986) specified the components of the educational system that 
contribute to student outcomes (see Figure 2). She indicated that the educational 
system consists of inputs, processes, and outputs (i.e., outcomes). Inputs included the 
human and financial resources that are available to education, while processes refer 
to the content and quality of instruction. Outputs were defined by Oakes as the 
consequences of schooling for students from different backgrounds. In Oakes' model, 
fiscal resources, teacher quality, student background and school quality are viewed 
as input variables. Curriculum quality, teaching quality, and instructional quality 
are seen as process indicators. Oakes differentiated three kinds of outputs or 
outcomes: (1) achievement, (2) participation, and (3) attitudes and aspirations. 

Models bv Shavelson and Associates 

Shavelson et al. (1987) developed a possible indicator system for use in 
mathematics and science (see Figure 3). In this model, they included input, process, 
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Figure 1. Carroll's Model of Learning (1963) 
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Teaching 
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Quality 
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Participation 
Dropouts 



Attitudes 
Aspirations 



Figure 2. Oakes' Comprehensive Model (1986) 
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Figure 3. Shavelson et al.'s Possible Indicator System (1987) 
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and outcome indicators, and linked each to current NAEP data collection efforts. In 
this model, outcomes are delineated more than in other models. Student achievement 
includes mathematics achievement and science achievement of all students, of 
college-bound students, and of prospective science/math majors. Student 
participation includes extracurricular activities and current math/science course- 
taking. Student attitudes include interest, liking, etc., social usefulness, career 
relevance, intended college major, and conceptions of math/science. 

Shavelson, McDonnell and Oakes (1989) used the same conceptualization of the 
educational system as did Oakes (1986), but pictured it as shown in Figure 4. This 
model designates inputs, processes, and outputs, with the same breakdown of outputs 
as presented by Oakes. 

National RRC Panel Framework of Effectiveness Indicators for Special Education 

The National Regional Resource Center (RRC) Panel created a framework of 
effectiveness indicators for special education (see Figure 5). They designed the 
framework to be compatible with the indicators model developed by the Council of 
Chief Stale School Officers (CCSSO). Effectiveness indicators for both general and 
special education were incorporated in order to clarify the relationship between the 
two systems. While the CCSSO framework was designed to generate data on the 
national level, the national RRC Panel created a compatible model that can be used on 
a district and local school level. The RRC Panel indicators are organized into six 
categories: (1) philosophy, policies, and procedures, (2) resource allocation, 
(3) staffing and leadership, (4) parent participation and community and 
interagency involvement, (5) instruction, and (6) program and student outcomes. 
An additional category of independent variables include "contextual factors." The 
panel chose a broad array of input, process, and outcome indicators in order to 
provide all the necessary elements for program evaluation and special education 
program improvement. 

Alberta Model of Student and System Outcome Indicators 

Alberta's Educational Department developed an outcomes model to "provide the 
uest possible education for al! Alberta students, including exceptional students" 
(Alberta Education, 1990, p. 46). The model (see Figure 6) is based on the belief that 
tiie school should always be the focus of effective education, and that schools have 
the sole responsibility for the development of students' competencies and school 
performance. The school, however, shares responsibility with other social 
structures for the personal/social and vocational student outcomes which are shown 
in the inner circle. The middle circle includes post-school outcomes in the areas of 
career development, social participation, and independent living. The level of 
satisfaction with special education that is expressed by the various stakeholders can 
be seen in the outer circle. 

The model represents the way outcome-based education can be evaluated and 
monitored to the satisfaction of all the partners. Education is perceived in the larger 
context of society and the responsibilities of each social entity are described. 

Conceptual Model s of Outcomes or Outcomes Indicators 

There is agreement that there ought to be a comprehensive model that 
underlies the development of a set of outcomes indicators. Yet, there is no agreement 




Figure 4. Shavelson, McDonnell & Oakes* Mode! (1989) 





Figure 5. National RRC Panel Framework 
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Figure 6. Alberta Model of Student and System Outcome indicators (1990) 
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on what the model should be. Burstein (1989) described well the dilemma we face 
when he discussed the range of possibilities in designing a model of outcomes 
indicators. As he says: 

The outcome domain in a national education data system is potentially 
indeterminate. For instance, outcomes could be restricted to those 
associated only with data cm or about students or also include teachers 
(professional qualifications, satisfaction), schools (climate, safety), 
community (attitudes and opinions) and so forth. The age span could focus 
on prccollegiate schooling or could include postsecondary and early 
childhood education as well. Even if restricted to students, the types of 
outcomes could include cognitive performance, school participation and 
holding power (school completion rates, attendance rates, grade retention 
rates, dropouts), attitudes, expectations, postsecondary transitions (college 
attendance, employment), and civic participation and responsibility (e.g. 
political participation). The boundaries separating outcome from 
curriculum indicators are also not well-established; conceptually, taking 
particular courses of study and particular courses that qualify students for 
various postsecondary opportunities is clearly an outcome, in part, of 
certain aspects of the schooling process. And, even if students are the 
ultimate source of the data, the outcomes of interest could be attributed to 
students, groups of students with certain characteristics, classes/teachers, 
schools, districts, states, etc. (p. 6) 

Hie complexity of the task is apparent from the multiple issues and possible 
perspectives that can be considered. The development of a conceptual model for 
outcomes that would direct the development of indicators requires even greater 
complexity of planning when students with disabilities are the focus of the model. A 
comprehensive model for special education needs to guide the development of 
indicators that reflect the following dimensions: (1) Content domains: 
competencies, skills, attitudes, behaviors, values, participation, contribution, and 
psychological functioning of students, (2) Developmental level: indicators that 
reflect outcomes of a variety of age groups, reference groups (e.g., family, peer, 
career, elderly), stages of development, and levels of schooling (e.g., preschool, 
elementary, secondary, post-school), (3) Severity of disability: indicators that 
reflect the conditions of disabilities as experienced by the individual and with regard 
to the degree of special assistance needed, and (4) Type of disability: outcome 
indicators that reflect the unique circumstances surrounding each disability (e.g., 
hearing, vision, retardation). 

A conceptual model of outcomes needs to further consider the relationship 
between the above dimensions (for special education) and all other indicators that 
are being used in regular education. It is important to conceptualize special 
education as an integral part of public education. Moreover, the educational system 
needs to be conceptualized within the context of our social and human environment. 
In this context, the role of various social institutions and their interrelationships 
needs to be considered in order to better understand the responsibilities and the goals 
of the educational system. 
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CcUteaJ fostiM to Consider in Developing a Concgplnfll Model 
of Outcomes and a System of Indicators 

The development of a conceptual model of outcomes and a system of indicators 
requires an understanding of every stage in the process of outcome assessment. 
Issues in four areas must be considered: (1) issues concerning the conceptual 
development of a model; (2) issues that pertain to the methodology of outcome 
assessment; (3) issues that are part of data analysis, interpretation, and policy 
making, and (4) issues that reflect the need to monitor decisions, change indicators, 
and revise the system of indicators over time. A recent paper by DeStefano and 
Wagner (1990) contributed significantly to the identification of many of these issues. 

Issues Related to the Conceptual Developme nt of an Outcomes Model 

The initial state of the development of an outcomes model should start with a 
collaborative effort to identify the assumptions held by stakeholders about outcome 
indicators. Agreement also should be reached on the definitions of the key concepts. 

Stakeholders' beliefs about important constructs in education and the nature 
of outcome indicators need to be identified. These beliefs probably should become 
p»rt of the assumptions of the model. Collaborative planning that begins with the 
identification of basic assumptions can "increase stakeholders 1 support and eventual 
use of outcome assessments and improve the design of the study* (DeStefano & 
Wagner, 1990, p. 16). 

Information collected during this initial stage also should include the 
identification of stakeholders' special needs and concerns, past and present practices ' 
related to outcome assessment (including the nature of national and state 
accountability systems and their operational methods), and the characteristics of the 
populations (i.e., students, professionals, administrators) in the social context (e.g., 
• school, family, community). 

All of this information can serve as the basis for the development of a 
conceptual model, which becomes a framework for outcomes assessment, 
interpretation, and policy decision making. Accordingly, the second stage in the 
development process should be the development of the conceptual model itself. 
DeStefano and Wagner (1990) emphasize that *a conceptual framework provides a 
structure for understanding, interpreting, and manipulating outcomes and should be 
specified in detail" (p, 19). The conceptual model needs to enable the understanding 
of outcomes in the larger context of the educational system and society, Thus, key 
independent variables, such as demographic variables, disability characteristics, 
household and community characteristics (DeStefano & Wagner, 1990) need to be 
included, and their expected influence on outcomes should be specified. After the 
context has been conceptualized, the specific outcome domains need to be identified. 
While many efforts in the past two decades have been focused on achievement 
testing, researchers and educators (e.g., Burstetn, 1989; Creech, 1989) recently have 
emphasized the importance of student participation and access, student status after 
completion of secondary school, and student attitudes, expectations, and aspirations. 

Accordingly, outcome domains, such as achievement, participation, and 
attitudes (Shavelson et ah, 1987) can serve as the major constructs of the model. 
However, within each domain (e.g., achievement), specific objectives need to I s 
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identified (e.g., content or subject matter mastery, high order skills). Snow (1989), 
for example, claims that learners 9 general objectives, across all subject areas, should 
be the development of conceptual structures, procedural skills, learning strategics, 
self-regulatory functions, and motivational orientations. For national and state 
levels, outcomes must be kept broad enough to function in a generic model, rather 
than to meet the needs of a particular group and specific contextual characteristics. 
The federal, state, and local roles in outcomes assessment need to be recognized at this 
stage of the planning. Elliott (1989), for example, believes that the federal role is to 
monitor achievement of national and state level goals, and to establish standards for 
data quality, content development, and data definitions. Furthermore, it is the federal 
responsibility to support research on factors that can facilitate the achievement of 
the various goals, and the overall performance of the educational system. Individual 
student goals, however, should afli be assessed on a national level Rather, the 
particular state and local circumstances should be taken into account in the 
establishment of outcomes measurement procedures. The conceptual model, 
therefore, should specify both general outcome constructs and specific outcomes in 
each domain. The relationships among the outcomes, the constructs, and the 
independent variables should be hypothesized as well. NCEO is committed to the 
development of the outcomes model and the system of indicators through a 
consensus-building process in which experts and stakeholders participate. 

Issues Related to Methodology 

Information gathered during the development stage, along with the 
conceptual model, can guide the planning of an appropriate methodology for the 
assessment of outcomes. Broad considerations related to methodology should be 
addressed first. It is important to delineate the purpose of data collection, the types of 
comparisons to be made, and subsequently, the nature of data collection (e.g., survey 

r census). Hie basic assumptions and the conceptual model agreed upon in the 
development phase should help define the purpose of data collection. If f for example, 
the model is to provide information only on the national level, this should direct the 
types of data selected. Moreover, the specific uses of the data at any level (i.e., 
national, state, district, school) need to be understood. On the national and state 
levels, for instance, data can be used for descriptive or comparative purposes. 
Descriptive statistics can provide general information such as national graduation 
rate or dropout rate. However, outcome indicators often provide littie information if 
they cannot be used in comparisons. For example, knowing that 32% of students with 
disabilities dropped out of school is more informative when we can compare it to the 
dropout rate of another group (e.g., students without disabilities). According to 
DeStephano and Wagner (1990), when considering special education, four possible 
comparisons can be made (1) comparisons of the outcomes of students in special 
education with the outcomes of students in regular education, (2) comparisons of 
students in different disability categories, (3) cross-unit comparisons, such as cross- 
state or cross-program comparisons, (4) comparisons of the same group or the same 
phenomena over time (as in longitudinal or time-series comparisons). Each of those 
comparisons can be further divided into comparisons of outcomes of different groups 
to a pre-established criterion (as in criterion-referenced assessment) or to a pre- 
established norm (as in nom-refcrenced assessment). 

While considering the desirable uses of the data, it is important to also consider 
desirable characteristics. Among these are thai (1) the validity of the comparisons 
will be assessed; (2) the demographic differences between the compared groups can 
be controlled; (3) outcome variations within each subgroup (e.g., specific disability 
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category) are incorporated; and (4) philosophical, political, economical, and cultural 
variables that affect the results are considered. Decisions about the purpose of the 
data collection and the type of comparisons to be made in turn influence the decision 
about whether the outcome measures should be in a survey or a census form. "A 
sample survey can be used if information is wanted only about aggregations of 
students; samples can provide information on the performance of schools, districts, 
states, and the nation. If information is needed about individual students to provide 
diagnostic information for their individual educational program, then every student 
must be tested, in the form of a census" (National Center for Education Statistics, 
1990a). 

* 

After clarifying the issues regarding the target population (e.g., national, 
states, etc.). the purpose of the data collection and the nature of the desired 
comparisons, a more detailed planning of the assessment procedures end processes 
can begin. Assuming that not every educational outcome for students with 
disabilities can and should be assessed in a census form, the next step would be the 
design and selection of a sample. Five sampling issues should be addressed before a 
decision can be made (DeStefano & Wagner, 1990). First, there is a need to study the 
"characteristics, bounds, strata, and unit" (p. 32) of the target population (i.e., 
students with disabilities), as well as to specify the comparison group(s). Second, it is 
necessary to consider sample size in order to enable generalizations to the whole 
population, and to detect significant between-group differences. Third, sampling 
requires that certain sample selection methods be used. "Subject selection can be 
accomplished by: (1) accessibility, (2) judgment, and (3) probability. Of the three, 
probability sampling increases the likelihood of sample representativeness. 
Individual, cluster, or stratified random sampling are the most commonly used 
probability sampling strategies" (DeStefano A Wagner, 1990, p. 37). Sampling errors 
need to be documented and later reported as part of the outcome study. Fourth, it is 
important to locate the respondents and verify that desired data are possible to 
obtain. For example, when individuals with disabilities are no longer in school, 
information about their location is harder to find. In order to avoid low response 
rates, location information can be found through pilot studies. Fifth, the actual 
obtained sample might end up differing from the planned sample. Therefore, it is 
necessary to document the sources of potential bias in the data and their effect on 
generalizability. Later on in the process, if bias was found, "statistic adjustments may 
be used to correct for differences or limitations may be placed on interpretations" 
(DeStefano & Wagner, 1990, p. 45). 

After considering the issues related to design and sample selection, it is 
possible to proceed to the next phase: selecting and opcrationalizing outcome 
measures (DeStefano & Wagner, 1990). At this stage, it is especially important to use 
the conceptual model for decision making. If, for example, the model includes both 
academic and nonacademic skills (e.g., independent living skills, social integration), 
the choice of measures should reflect ail these facets. It is also necessary to 
reconsider the available and commonly used indicators of outcomes (e.g., grades, 
attendance rate, suspension, achievement/competency, school completion status, 
employment, post-school education, quality of life). Professional evaluators (e.g., 
Cronbach et ai. 1980; Glascr, 1988; Shcpard, 1989), however, argue that there is a need 
for belter assessment measures. They acknowledge that standardized tests (e.g., 
minimal competency testing, SAT, ACT, NAEP) do not cover the full range of 
important instructional objectives, and that these tests are overly used for purposes 
they do not fit. Innovative measures might need to be developed in order to measure 
the new and the existing domains in different ways (e.g., more valid, accurate, 
effective). 
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While considering the measures, it is important to acknowledge the limitations 
of the selected measures and the implications of the choice to comparison and 
interpretation purposes. The validity of each measure can be thought' of as the 
appropriateness of the inferences that can be made from the measure (derived from 
the 1974 APA definition of "validity"). Validity, then, cannot be only the product of a 
consensus process. Valid measures need to be supported by empirical evidence as 
well. It is further important to consider the type of data that tt measure would 
provide (i.e., qualitative or quantitative), the kind of assessment ol that would be 
used (e.g., a test, a survey, a questionnaire) and whether the appropriate reference 
information (i.e., criterion-referenced or norm -referenced) is available. Voeltz and 
Evans (1983), for example, argue that since we do not have "normative data regarding 
the growth that can be expected" (p. 4) from the various groups of learners with 
disabilities, we currently do not have the proper measurement procedures for 
grading, promotion, graduation, and other evaluative activities. Those who choose 
measures for assessing outcomes in special education need to be aware of all these 
issues and their implications for interpretation and decision-making. 

The last stage in the planning process includes the consideration of four 

issues: 

# 

(1) Choosing the data sources for the different outcome measures. School 
records and personal reports from adults (e.g.. parents) or youth are 
possible sources. 

* 

(2) Data collection methods need to be selected. Common methods are record 
review, questionnaires, portfolios, interviews, tests, and observations. 
Choices of both data sources and collection methods need to be made, 
addressing the limitation of each choice (e.g., availability, access, cost, 
etc.), and its contribution to the quality of data interpretation. 

(3) At times, there might be a need to develop original instruments for 
assessment. 

(4) It is necessary to determine the timing of the data collection. For 
example, should data be collected for grades four, eight, and twelve, or 
any other grade? How soon should measures be collected for people who 
left school. How often should measurement occur (every two years, 
every five)? Questions such as these need to be answered in order to 
enhance the validity of the information and the feasibility of 
maintaining the chosen intervals. 

Issues Related to Pata Analysis, Interpretati o n, a nd Policy Mating 

Considerations of appropriate analysis methods should start in the early 
process of methodological planning. Planning early can prevent a situation where 
the characteristics of the collected data (e.g., sample size, level of measurement) do 
not enable us to answer important questions because only certain types of analyses 
can be used with this type of data. Planning for analysis requires the consideration 
of several issues (DcSiefar.o & Wagner, 1990): 

(1) Understanding the nature of the outcome information that is needed, 
the specific sample, comparison groups, dependent and independent 
variables, and the relationships of interest among them. 
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(2) It is important to understand the level of measurement (i.e., nominal, 
ordinal, interval, and ratio) of the dependent and independent 
variables. The level of measurement is important in the selection of 
analysis method (eg., analysis of variance, discriminant analysis, factor 
analysis, etc.). 

(3) The sample size and the availability of comparison groups can affect the 
choice of the analysis method. Some types of analysis (e.g., factor 
analysis) require a large sample size. In other cases, missing data can 
make the remaining sample less representative of its population. 

(4) "The knowledge base and experiences of the audience should not dictate 
the choice of analysis, per se, but should be considered when deciding 
how to report findings and disseminate results" (DeStefano & Wagner, 
1990, p. 83). 

Communication of outcome information has to be done in a way that 
acknowledges multiple audiences. Practitioners and policy makers, for the most part, 
are interested in questions and answers, not Specifically in the data. Narrative and 
graphic displays may prove most informative for them. Therefore, presenting the 
data is not sufficient Researchers probably will want to interpret the findings and 
discuss . all they have learned, even if that means presenting multiple 
interpretations. Regardless of the approach, the limitations and the shortcomings 
(e.g., small sample size, measurement error, etc.) need to be discussed, along with the 
way they affect the validity of the findings. 

The existence of multiple interests and audiences requires that several formats 
of dissemination be used. While a lengthy report (i.e., "Final Report") might be 
useful for present and future researchers, brief reports, such as journal articles, 
newsletters, and video presentations can be prepared to inform practitioners, policy 
makers, and the general public. 

Issues Related to Monitoring and Changing the Model 

The current reform movement and its emphasis on results and accountability 
make outcomes assessment a necessary element in evaluating, planning, and 
monitoring the educational system. Making outcomes assessment a routine part of 
the accountability systems of states and the nation creates a need not only to change 
aspects of the educational system according to the outcome information, but also to 
change the model to fit the changing reality. 

It is clear that the development of an outcomes model and system of indicators 
cannot be viewed as a one-time effort. The dynamic nature of the many variables 
that are pan of the model create a need for continuous development. Some of the 
changes that would press for revisions of the outcomes model include new 
educational goals (or new understandings of the same goals), new assessment tools, 
different ways to analyze and interpret data, and new social realities (that reflect 
cultural, demographic, economic, or other sociological changes). Therefore, since 
these changes are an integral and inevitable part of the educational system, it is 
necessary to create a context of change. In this context, where the model is in a 
continuous process of dynamic development, decisions should be made about who 
should be responsible for producing outcome data and what should be the nature of 
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the revision process. There is a need to understand the change process for revision 
of the model, including who should decide what needs to be changed, what should be 
the preferred course of change, its timing, and its impact on the educational system. 

Accepting outcomes assessment as a routine requires the states and the nation 
to decide where the resources for it are to come from and how the process can be 
communicated in an effective way. Students with disabilities, and the field of special 
education in general, can benefit from outcome information if the model reflects a 
continuous effort to respond to information needs with valid and reliable data, with 
sensitive policies, and with a dynamic approach for educational improvement. 
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Activities related to the development' of educational indicators, particularly 
indicators of outcomes, are moving along at increasingly rapid speeds. New groups 
are formed and new meetings held almost on a daily basis. Most of the activity that 
has occurred has been in the general education sector. Yet, progress is being made 
also within special education, and to some extent, in attempts to merge general 
education and special education activities. In this section, we review the major 
policy groups involved with outcomes indicators and the nature of their activities. 

The description "policy group" is used broadly here to reflect any group, other 
than states, attempting to make statements about policy related to educational 
indicators. Most of these groups are at the national level, but there are also several 
relevant regional policy groups that are intensively involved in discussing and/or 
developing educational indicators. In the next section of this paper we review the 
activities of state education agencies. 

In attempting to review what we know about the current status of outcome 
indicator activities, we have opted to begin by treating separately the activities of 
general education and special education policy groups. When this separation has 
been crossed (such as when a general education policy group addresses the needs of 
students with disabilities), it is noted. 

General Education 

For the novice, the myriad of organizations taking part in the game of 
developing educational indicators is overwhelming (see Table 5). In order to better 
understand current activities in this area, it is helpful to have a picture of the major 
players in the game, and their relationships to each other. An attempt is made here 
to do just that, focusing at this point just on national and regional policy groups. 

Major policy groups exist within the U. S. Department of Education, 
particularly the National Center on Education Statistics (NCES). Two significant 
players within NCES were established under authorization of the Hawkins-Stafford 
Act: (1) the National Forum on Educational Statistics (often referred to as the Forum), 
particularly one of its committees, the National Education Statistics Agenda Committee 
(NESAC), and (2) the Special Study Panel of Education Indicators (often referred to as 
the Indicators Panel). 

The National Forum on Educational Statistics (Forum) is an independent 
body whose mission is to propose and support improvements in the Federal-State 
partnership that collects and reports elementary and secondary education statistics. 
The Forum was legislatively mandated by the Hawkins-Stafford Education 
Amendments of 1988 (P.L. 100-297). The goal of the Forum is to influence the 
production of a national cooperative statistics system of useful and comparable data. 
Members of Forum are representatives of ail .States, territories, major federal 
education departments, and national educational organizations. The forum is divided 
into five committees. One of these, the National Education Statistics Agenda 
Committee (NESAC) servt.s to guide the entire Forum and future policy toward 
"education-relevant" agenda items. In December, 1990, NESAC released a report 
entitled, A Guide to Improving the National Education Data System. The report 
included 36 recommendations for improving national data in the area of 
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demographics (7 recommendations) resources (12 recommendations), processes (6 
recommendations), and outcomes (11 recommendations). With input from the Office 
of Special Education Programs, included are several recommendations that 
specifically address the needs of students with disabilities. 

The Special Study Panel on Education Indicators (Indicators Panel) was 
also established under authorization of the Hawkins-Stafford Act Its purpose is to 
make recommendations concerning the future determination of educational 
indicators. The Indicators Panel is comprised of 19 members appointed by the 
Secretary of Education and is chaired by Alan Morgan, Governor of New Mexico. Its 
charter to address future -oriented issues ends with the publication of its report to 
Congress due in May, 1991. There are three designated workgroups within the Panel 
(A,B,C), each addressing selected issues (e.g., readiness for school, educational 
equality for children at-risk of school and societal failure, quality of schools and 
educational experiences, acquisition, appreciation of, and engagement in subject 
matter; advanced academic/thinking and citizenship skills, educational contributions 
to economic productivity— quality of the workforce, international competitiveness, 
issues of the labor market-and societal support for schools and learning). 

. In its guidelines for the Special Study Panel, NCES noted that educational 
indicators could provide information concerning: (1) outcomes (e.g., achievement, 
attainment, postsecondary experiences, beyond school experience), (2) input (e.g., 
resources, teacher quality, quality of curriculum), (3) process (e.g., attendance, 
instructional strategies, individual allocation of time, expectations, commitment and 
effort, support services, extracurricular activities, personalization, 'school climate), 
and (4) context (e.g., student characteristics, district/school characteristics). It was 
noted that the panel members should clarify the nature of indicators for use at the 
national level, for use as state-by-state indicators, and for use as district and school 
indicators. It was also emphasized that decisions on measurement should be tied to 
the educational goals, and should provide clear criteria for the interpretation and use 
of the data. Further NCES recommended that the Special Study Panel give 
consideration to future composite indicators that capture "a particularly significant 
aspect of schooling or an emerging area of policy interest" (Special Study Panel on 
Education Indicator \ 1990). Examples of areas for composite indicators included an 
index of "at-riskrrss" and a "gross national educational product." 

In February, 1990 the Special Study Panel on Educational Indicators held a 
working group meeting to structure "candidate indicators" acceding to logical 
groups and degrees of disaggregation (e.g., local, state, national). Other possible 
breakdowns included: (a) level of governance (preschool, elementary, etc.), (b) 
location (rural, suburban, urban), and (c) other demographic factors (race, SES). 
Indicator groups included: inputs, context, process, resources, demographics, and 
outcomes. The non-exclusiveness of some variables was also a key issue. For 
example, school climate may be considered a possible process or a possible outcome. 
The work focused on a conceptual level for each of the indicator groups. The 
working group raised several key concerns with respect to particular indicators. For 
example, if outcome data can be expanded to include nearly any measurable variable, 
at what point are enough data collected? Further, appropriate assessment (e.g., 
cross-sectional versus longitudinal) and instrumentation (methodology, analysis) for 
particular indicators was questioned. Finally, the group stressed that a core group of 
indicators common to all levels of disaggregation (once agreed upon) would be 
important to provide a compatible connecting data base. A long-term commitment to 
the indicators that are adopted was seen as essential. 
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The meeting of the Indicators Panel that was held in December, 1990 focused 
on the. report scheduled for release in the Spring of 1991. Currently, thi document is 
planned to have chapters on the topics of readiness for school, quality c schools and 
educational experiences, societal support for learning, youth's engagen cnt in 
learning and acquisition of knowledge and skills, educational contributions to 
economic productivity, equity for children at risk of school and societal failure, and 
components of an ideal indicator system. A draft of the last chapter did not exist at 
the time of the meeting. 

NCES has five divisions, all of which are relevant to the activities of the 
National Center on Educational Outcomes. The two groups authorized by the Hawkins- 
Stafford Act are part of the Special Surveys and Analysis branch of the Division of 
Elementary and Secondary Education Statistics. Other branches include General 
Surveys and Analysis (which includes the National Assessment of Education 
Progress-NAEP) and the Longitudinal and Household Studies (which includes the 
National Longitudinal Study 72--NLS, High School and Beyond-HSB, and the National 
Educational Longitudinal Study, 88--NELS). 

The current vehicle for national and state-by-state assessment of educational 
indicators is the National Assessment of Educational Progress (NAEP), administered 
since 1983 by Educational Testing Service (ETS). While NAEP has monitored trends in 
U.S. education since 1969, it was not until 1983 that the focus of the assessment was 
defined to be information relevant to policy makers. In 1988. Congress ratified a two- 
stage trial by NAEP of voluntary participation of states in assessments using 
comparable indicators. Many interacting forces led to this trial assessment 
(scheduled for 1990 and 1992). For example, the Council of Chief State School Officers 
(1987) wanted state-by-state education indicators and the National Governors' 
Association (1986) had called for better report cards. Furthermore, there was the 
recognition that the "wall chart" provided shaky indicators of cognitive outcomes 
(since it relied on average scores of college-bound high school seniors on the 
Scholastic Aptitude Test and the American College Test). 

As suggested above, two policy groups external to the U. S. Department of 
Education have played a major role in generating interest in developing educational 
indicators: the National Governors' Association (NGA) and the Council of Chief State 
School Officers (CCSSO). NGA was involved in the initial development of the goal 
statements and is now preparing a plan and timeline for reachin* the goals. CCSSO 
has many projects relevant to educational indicators. Perhaps most directly relevant 
is the Science-Mathematics Indicators Project directed by Rolf Blank (1987). This 
project is developing state-level indicators of key dimensions of elementary- 
secondary education in science and mathematics. A plan has been proposed for a full 
set of state-level indicators to include program inputs and achievement and other 
outcomes. Others at CCSSO are working with NCES on the Data Improvement Project 
and internationally with projects on international indicators in education. 

After the national goals were announced by the President and the governors, 
a panel was established to issue a "national report card" on education. The National 
Education Goals Panel (NEGP), led by Governor Roy Romer of Colorado, is made 
up of six elected governors and four administration officials. Recently, U.S. 
legislators have called for the establishment of another group to work on assessing 
the national goals. Essentially, the amendment was offered by Senator Jeff Bingaman 
to allow educators, representatives of the business community, and parents to provide 
input into the assessment of education, and would have designated $2 million for a 
National Council on Educational Goals. In January 1991, the National Education 
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Goals Panel announced that it had created five resource groups that will advise it on 
options for its annual report card to the nation, the first of which is scheduled to be 
released in September, 1991. The groups, which include experts in educational 
assessment and measurement in specific goals areas, focus on the goal areas of (1) 
school readiness, (2) student achievement, (3) mathematics and science achievement, 
(4) literacy and life-long learning, and (5) safe and drug-free schools. Whether this 
additional component to the National Education Goals Panel would satisfy the 
legislators remained to be seen. 

The National Center on Education and the Economy, in conjunction with the 
Learning Research and Development Center at the University of Pittsburgh, received 
twin grants of $1.15 million and $1.3 million to develop a new national 
examination system for students. This system, which is geared toward high-level 
skills and the application of knowledge to real world problems, is currently 
conceptualized as a series of performance examinations, portfolios, and projects that 
students would complete over time in order to graduate from high school. Individual 
states or school districts would be able to develop their own instruments for 
assessment; these would be calibrated to national standards in core subjects. 

Implications of general education outcomes activities f or students with 
disabilities. With all that is going on in general education, it is important to ask 
about the implications these activities have for special education. For the most part, 
limited attention has been given to the special learning and adjustment needs of 
students with disabilities. This problem is app<u~nt in limited accommodations in 
assessment practices (e.g., for sensory disabilities), in the absence of educational 
goals and indicators, and in the limited areas of assessment and usefulness of data. 
The renewed interest in education during the 1980s that triggered the reform 
movement in general education often excluded students with disabilities from the 
debate. The major reforms neglected to discuss the issues that affect these students' 
educational condition. Reports such as Children in Need (Committee for Economic 
Development, 1987), which discussed the problem of students "at risk," did not include 
students with disabilities in their reform initiatives. The Council of Chief State 
School Officers briefly recognized students with disabilities in their proposal to 
"guarantee those students least likely to graduate from school access to quality 
education programs" (Council of Chief State School Officers, 1987). 

Legislation for the education of individuals with disabilities indicates, 
however, that there is a growing public interest in making the issue of improving 
the assessment of educational outcomes for children and youth with disabilities a 
major agenda item in the coming years. The Education for All Handicapped Children 
Act (PL 94-142) is a case in point. More recently, at the Educational Summit Meeting 
in Charlottesville, President Bush noted that there is a need to ensure that "no child 
in America be forgotten or forsaken" and that "...this includes both the unusually 
gifted and those with special needs and disabilities. But it must also include the 
student too often forgotten, the 'average student'" (September 28, 1989). Accordingly, 
the reform movement that calls for accountability by means of accurate assessment 
of outcomes should include all students. Indicators should be developed to study the 
condition and improvement of education for students with disabilities as pan of the 
broader educational indicators movement. Yet, this may not be happening. 

For example, in the NAEP system, the assessment guidelines (Mullis, 1990) note 
that although it is NAEPs intent to assess all selected students, "some students sampled 
for participation in NAEP arc excluded from the sample according to carefully 
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defined criteria" (p. 35). Among those who "may be incapable of participating 
meaningfully in the assessment" (p. 35) are many students on Individual Educational 
Plans (DSPs). These students may be excluded if: "The student is mainstrcamed less 
than 50 percent of the time in academic subjects and is judged to be incapable of 
taking part in the assessment, or the IEP team has determined that the student is 
incapable of taking part meaningfully in the assessment" (Mullis, 1990, p. 36). In 
reality, despite the NAEP admonition to include a student if there is doubt, schools 
may be excluding these students whenever possible, in part to spare the student the 
stress of this testing experience and in part to raise the district's average score. A 
specific caution mentioned in connection with the NAEP data is identified by Phillips 
and Finn (1990): 

We cannot generalize to populations of students for which the sample was not 
designed to produce estimates. Hie sample contains some students who are 
handicapped, limited in iheir English proficiency, enrolled in vocational 
education programs, and receiving Chapter 1 services. Yet the state samples 
are not designed to be representative of these groups. Hence, state estimates 
cannot be supplied for the achievement of these subpopulations. (p. 52) 

The issue of exclusion appears to be one that will rise again in other national 
data bases (such as NELS 88) and in data bases still to be established (Adult Literacy, 
for example). 

A recent survey of the 50 states and the District of Columbia (Bodner, Clark. & 
Mellard, 1987) examined several issues that become relevant when students with 
disabilities are held accountable for state educational requirements. Bodner et al. 
(1987) found that state rwlicics and planned policy changes hold important 
implications for students in special education. For example, 36 states had increased 
graduation requirements since implementation of PL 94-142, and 36 states (not 
necessarily the same ones) had specified that some sort of allowances be made in 
meeting graduation requirements for students in special education. Thirty-one 
states have policies determining exit documents (e.g., diplomas) by the state 
education agency; the remaining states allow the local districts to make such 
determinations. Seventeen states require that differential exit documents be awarded 
to students in regular and special education. 

Minimal competency testing (MCT) is the source of another Important issue. 
Thirty states have some u-rt of MCT program, but only 23 have written MCTs. Of these 
23 states, 21 require the test be taken and 22 have a specific policy or practice 
regarding students in special education. These policies usually take the form of an 
accommodation for students, such as individualized administration, extended time 
periods for testing, or having the test administered by a special education teacher. 
In 15 states, the students' MCT scores determine the exit documents they will receive. 
Some states specifically exclude students with mild disabilities from their norms and 
published results. 

Minimal competency testing has a significant impact on students with 
disabilities when used to determine graduation eligibility or the type of exit 
document. Some years ago, the U.S. Department of Education (Wildcmuth, 1983) 
outlined six "accommodating" strategics: exclusion of children with disabilities from 
MCT programs, substitution of IEP goal attainment, establishment of differential 
standards, awarding of differential diplomas for students not taking or passing the 
MCT, modification of test administrations, and use of different minimal competency 
tests. Pullin and Zirkcl (1988) reviewed and summarized the legal ramifications of 
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MOT policies, noting this activity and presenting two major conclusions: "(1) If 
sufficient accommodations in test administration to students with handicaps have 
been made, students may be denied receipt of a regular diploma when a new testing 
requirement is used even if they meet ail other standards; this is not a violation of 
Section 504 of the Rehabilitation Act of 1973; and (2) There must be adequate advance 
notice to allow sufficient coverage of the skills and knowledge covered on the test in 
a student's IEP, or for the parents and teachers to make an informed decision that the 
IEP not be geared to the test" (p. 14). 

More recently, Vitcllo (1988) summarized MCT policies for students with 
disabilities in the SO states. He found that most states address the issue by waiving the 
requirement. These data raise issues about minimal competency testing and its often 
paradoxical relationship to individualized educational planning. Again, the balance 
between appropriate educational programming and equal opportunities in education 
and access to a diploma is put in question. So far, the reform agendas stressing 
outcomes created an environment of exclusion from testing, evaluation, and 
graduation certification for students with disabilities. Policy makers in general 
education have not yet addressed the rights, the needs, and the appropriate ways to 
include students with disabilities in outcome-based education. 

While the implications of general education reforms for students with 
disabilities have been generally negative or exclusionary, positive approaches to 
reform have occurred in special education. It is important to look at what has been 
done so far in the way of developing special education indicators. 

. Special Education 

While the Office of Special Education Programs (OSEP) has attempted to 
influence policy in general education as activities in that realm exploded, it was not 
until the National Center on Educational Outcomes was established in October 1990 
that there was a national entity devoted to identifying outcomes and indicators for 
students with disabilities. 

On a regional basis, however, considerable activity had occurred through 
Regional Resource Centers, which had established a National RRC Panel on 
Effectiveness Indicators for Special Education. This panel produced a reference tool, 
"Effectiveness Indicators For Special Education" (available from the Mid-South 
Regional Resource Center), which has been a take-off point for many states involved 
in developing indicators outcomes. 

In late 1990, the National Council on Disability (NCD) requested 
proposals for projects to study the outcomes of elementary and secondary educational 
programs for students with disabilities. In its request, the Council identified the 
following as outcomes: 

• Academic achievement 

• Work readiness 
Improved quality of life 
• Internal factors 

• Job satisfaction 

• Satisfaction with social relationships 

• Optimism and hope for a better future 



• Good coping and communication skills (reading* writing, listening, 
and speaking) to deal with the problems of daily living 

• Self esteem 

• Improved quality of life-continued 
• External factors 

• Sufficient income to meet the needs and demands for independent 
living and good health 

• Clean, sanitary, and comfortable housing 

• Community support systems to provide a safety net when difficulties 
arise specific to the person's disability 

• Identification with advocacy groups to guarantee the civil rights for 
persons with disabilities 

• Efficient and satisfactory transportation 

• Evidence of involvement in community activities, especially those 
that benefit others 

The 1989 NCD report directed to the President and the Congress of the United 
States, entitled "The Education of Students with Disabilities: Where Do We Stand?", 
followed the Council's own reports of 1986 (Toward Independence) and 1988 (On the 
Threshold of Independence), which suggested the need to establish a national 
commission to examine the quality of education for students with disabilities in 
America. The latest report argues that "the time has come to ask the same questions 
for students with disabilities that we have been asking about students without 
disabilities" (p. 2). Included among the questions that the Council suggests we ask are 
questions related to student achievement and participation while in school (Are they 
achieving? Are they staying in school?), and questions about their readiness to 
move to new activities (Are they prepared to enter the work force when they finish 
school? Are they going to participate in postsecondary education and training? Are 
they prepared for adult life?). While these kinds of questions are the beginning 
points for examining student outcomes from schooling, there are many broader 
issues to consider as well as specific ones for students with disabilities. 
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General Education 

Summaries of activities. School reform agendas also have spurred a variety of 
activities in states with respect to indicator systems and outcomes assessment in 
general education. Table 6 is a visual summary of the results of a project (Inman, 
Prcbish* & Sal gam k, 1990) designed to provide information on state-level activities in 
these areas. The groupings of variables into categories (e.g., student progress, 
attitudes) are our own, and some of the definitional difficulties are reflected in the 
table. Note, for example, that dropout rate is placed under Student Participation, and 
graduation rate under Student Progress, though both represent exit data of some sort. 
Student Participation and m Student Progress measures are clearly the predominant 
data types currently collected and reported. Nearly all states collect student test data 
at some point in the school careers of children, yet, the specific nature of these data 
(criterion or norm-referenced, etc.) varies considerably among ihe states, and 
certainly among local districts. 

It can also be seen in Table 6 that there are great differences among states in 
the scope and range of data collected. Some states (e.g., California, South Carolina) 
report collecting data on many variables in all groupings, while others collect data 
on only a few variables. In a report prepared for 4he Council of Chief State School 
Officers* Selden (1990) noted that the indicator systems reflect the level of state 
investment in educational programs and professional development Overall. He also 
noted that state indicator systems "reflect individual histories and circumstances of 
the state — particularly the balance of state and local control of policy making" 
(p. 6) In this report entitled "State Indicator Systems in Education," the indicators are 
classified according to a taxonomy consisting of Content, Level, and Policy/Purpose. 
Outcome measures are included under Content, and are referred to as "end products," 
Selden reports that "educational outcome data are clearly the most prevalent category 
of indicators to be reported by states* (p. 9). He mentions test data, graduation rates, 
dropout rates, and tracking after school as common examples, but suggests that other 
measures such as teacher attendance, parent perception of schools, and student 
perception of schooling are also considered to be outcome data by some states. In 
fact, Selden points out that "the main controversy among state systems is where to 
include some variables." (p. 12). Teacher and student attendance, for example, are 
outcomes (end products) in some states, but considered process variables in others. It 
is generally difficult to find consensus among the states on the specific classification 
of variables, but Selden lists only Alabama, Montana, South Dakota, and Tennessee as 
not reporting some sort of program indicators or statistics on educational quality. He 
also notes that these are mainly general education based data, and only "a few states 
factor in numbers of students served who are handicapped" (p, 10). 

The Association of State Assessment Programs (ASAP) produces semi-annual 
reports (Roeber, 1990) on the states* large sca!e assessment programs. ASAP is a 
private organization of state testing directors, supervisors, and other personnel, 
started about 10 years ago when Ed Roeber and Jim Fisher (FLA) saw a need for people 
around the country to have a network of information exchange available to them. 

A survey is the vehicle for this network, and is now done two times per year. 
In October, a complete survey (about 14 questions) is mailed to each state's director of 
testing. Participation is strictly voluntary. In March, a shortened version of the 
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questionnaire is mailed. The questions included are mainly about what new things 
states are doing. Responses to this version ar considered more "public* and are 
usually presented at the Education Commission of the States' June meeting. The 
report produced by Roeber is very detailed,' including listings of answers to each 
question by individual states. Questions cover such topics as the instruments used at 
which grades, number of students tested, sampling methods, etc. One question asks 
specifically about the state's policy for special needs students (Individualized 
Educational Program and Limited English Proficiency students) concerning high 
school competency testing. 

In the following paragraphs are summarized some of the specific state 
activities. Two of the more established and commonly mentioned state-driven 
indicator systems are those of New Jersey and California. In addition to these states, 
the Departments of Education in Kentucky, Vermont, and Florida have recently 
approved major changes in their systems of education. These are summarized 
because they have been identified by press articles as states actively involved in 
reform. Other states may also be engaged in changes or contemplating them in their 
systems of education. 

New Jersey. The New Jersey State Department if Education is often cited 
because it has the power to enforce its standards, and es indicates ,u ....... 

compliance. The state awards district certification if data collected on 10 approved 
indicators are found to be acceptable. In addition, at the ool '-".-!, New Jersey 
emphasizes outcome data such as test performance, attendance, SAT ^ 

California. California uses the independent Polic; Analysis for California 
Education (PACE), as a data summarization design, to keep rack of a wide variety of 
educational indicators. The California system is very complex, and has been 
frequently cited as a distinctive state approach to promoting excellence (Murphy, 
Mesa, & Hallinger, 1984; Timar & Kirp, 1989). The state plan involves using financial 
incentives, but with very little control over reform programs, while collecting 
indicator data from a variety of sources (not just the Department of Education). 
Currently, California has an established accountability system and collects data on 
over 40 performance indicators. An index called the California Accountability Index 
(CAI) (California Department of Education, undated), analogous to a Dow-Jones 
Industrial Average, is being proposed to measure performance with a . single 
accountability index. The CAI has 15 indicators divided into four clusters: 
achievement, college bound, dropout, a. i placement. Each indicator measures the 
percent of students who meet or exceed i set criterion on that variable (e.g., a 
standardized test score). The CAI has a nean of 1000 and standard deviation of 100; it 
assesses the "percent above" certain levels on the indicators. Grade-level CAI's are 
possible, but overall CAI's are typically reported by districts. Comparisons can be 
made at school, district, and state levels. 

Kentucky. In late March, 1990, the legislature of the State of Kentucky 
approved a massive school reform bill geared toward performance-based outcome 
assessment for all of the state's schools. The bill contains seven capacities and goals 
drawn from those established by the National Governors' Association, covering ail 
aspects of child development from • academics to physical and mental well-being. The 
legislation addresses needs from early childhood education through post-secondary 
programs, and includes students of all ability levels, from students with disabilities to 
students considered gifted. The inclusion of children with disabilities represents an 
important facet of the reform, for it gears the performance-based assessments, 
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Table 6 



Summaiy of State Activiiios on Indicator Systems a 



1 2 3 
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Participation 

4 5 6 7 8 9 10 
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2 3 4 5 6 7 



Studont Post* Fiscal Staff 
School Data Data Data Attitudes 

1 2 3 4 1 1 2 1 2 



Administrative 
Data 

1 2 3 4 5 6 
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Table 6 (continued) 



Student 
Participation 



Student 
Progress 



Student Post* 
School Data 



Fiscal 
Data 



Staff 
Data 



Administrative 
' Data 




3 Derived from inman, Prebish, & Salganik (1990). 
•These states did not provide Indicators 

Student Participation 

1 « Demographics/S£S 

2 - Preschool involvement 

3 m EnrollmonWronds 

4 - Attendance/absenteeism 

5 m Studont mobility 

6 - Academic course enrolimonl 

7 • # or % receiving special sorvicos 

8 - Suspension/expulsion 

9 * Dropout rate 

10 - Vocational education 

Student Progress 

1 - Test data 

2 - Language proficiency 

3 « Retention rales 

4 » GPA 

5 ■ Graduation rate 

6 « Evaluation of c ? ass activities 

7 » % Taking SAT/^CT 



Student Post-School D^ a 

t m % to college or postsecondary training 

2 m Performance in college 

3 - Tracking - postsecondary 
4-GED 

Fiscal Data 

1 - Per pupil or other fiscal data 

Slnff D ata 
t m Staff salaries 

2 » Stall altendance 

Attitude^ 

1 » Parent-student attitude 

2 - Climato - foadorship involvomonl 



Administrative Data 

1 - Graduate requirements 

2 m Instructional time 

3 m Curriculum-test match 

4 - Student-teacher ratio/class size 

5 - Alternative education programs 

6 - Amount of homework 
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which will be used as the basis for rewards and sanctions for schools, toward fair and 
representative evaluation of all segments of the population. The task of formulating 
measurable student outcomes for all students belongs to Kentucky's Council on School 
Performance Standards. The legislation provides for this group, and the Department 
of Education in general, to develop the new assessment techniques required under 
the new system. The process of formalizing the performance-based assessment of 
outcomes is expected to be accomplished by 1994. 

Vermont. Vermont has similarly adopted a new statewide evaluation system 
that emphasizes performance-based assessment Unlike Kentucky's system, 
Vermont's system focuses only on a student's academic competence. Vermont's 
system, currently in its pilot year, focuses on student "portfolios* or student files in a 
subject area that contain examples and best pieces of the student's competence. 
Reading and writing for children in grades 4-8 have been targeted for 1990-91, but 
Vermont plans to extend the system statewide and across all academic subjects. The 
portfolio system, developed by Vermont teachers, represents Vermont's attempt to 
broaden typical educational outcome measures. Students are evaluated on the basis of 
mastery and progress evidenced in their files, and schools are assessed by the 
composite profiles of their students. Vermont is defining outcomes, therefore, in 
terms of academic competence demonstrated by students. The system is designed to 
foster student responsibility for learning and provide data both on how a student 
understands and how effectively schools are fostering student competence in 
academic subjects. The portfolio assessment system represents another product of 
the push toward performance-based assessment. 

Florida. Florida has implemented a system of performance-based education 
since the mid-1980s. Florida's system is composed of minimum competency standards 
for students in five subject areas: math, reading, writing, career, and 
social/personal. The standards apply to all students in regular education and selected 
groups of students in special education. To receive a regular education diploma, 
students have to demonstrate mastery of ail the standards. However, any of the 
required tasks can be modified &vd adapted to meet a student's needs (e.g., exams can 
be administered orally for a student with a physical impairment). Separate standards 
apply to students in special education diagnosed as EMH (Educable Mentally 
Handicapped), TMH (Trainable Mentally Handicapped), Visually Impaired, or Hearing 
Impaired. Students meeting these adapted standards receive a special diploma 
signifying their mastery of special coursework. The standards, therefore, are also 
used to determine whether a student will receive a regular or special education 
diploma. 

Originally, Florida's Department of Education was charged with overseeing the 
implementation of these standards. Currently, however, local districts are 
responsible for implementing the standards, with state-wide monitoring every few 
years. In 1990, Florida's State Board of Education voted to replace the entire minimum 
competency standards with a new system, to take effect in 1994, that will emphasize 
educational outcomes. 

Special Education 

Summaries of activities . The National Association of State Directors of Special 
Education (NASDSE) sponsored working meetings in conjunction with the Office of 
Special Education Programs (OSEP) in the U.S. Department of Education to provide a 
forum for state education agencies to share information concerning issues, design, 
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and implementation of outcome assessment models for students with disabilities (June 
11-13, .1989). States invited to participate were Connecticut, Florida, Iowa, Michigan, 
Nebraska, New Hampshire, New York, and Oregon. 

As in general education, the same kinds of data were considered by some states 
to be indicators of effectiveness and by other states to be student evaluation data. 
States also varied on domains assessed, with some emphasizing academic outcomes and 
others stressing postschool experiences (e.g., employment, living arrangements). 
Technical issues of assessment were also reported by states to be important, but the 
political implications associated with technical problems worried the state education 
agencies even more. States are confronted with issues of inference and 
generalizability based on their data. A dominant issue of discussion was that of 
identifying the outcome indicators) that are appropriate for the particular 
question(s) of interest to the state. 

The influence of contextual factors throughout all phases of assessment is 
commonly presented as a problem in determining outcome indicators. Special 
education service delivery approaches, measurement issues, political pressures, and 
other factors interact in a variety of ways to affect the purpose, methods, and 
interpretation of indicator data collected by states. The composite summary included 
in the NASDSE report, and presented here aj Table 7, also shows the varying degrees 
of progress of these states in exacting outcome indicator assessment for students with 
disabilities. This variance can also be viewed as reflecting states' individual 
histories, philosophies and initiative concerning outcomes assessment of students 
with disabilities. 

Variance in state practices is reflected similarly in the doctoral dissertation of 
Chriss Walther-Thomas (1991) from the University of Kansas. Her thesis examined 
state-level practices regarding collection, use, and dissemination of outcomes 
assessment information on students with mild disabilities. She also was interested in 
describing relationships between the State Education Performance Chart variables 
used by the U.S. Department of Education and the types of outcomes data collected by 
states. 

Walther-Thomas (1991) found that there were no comprehensive outcomes 
assessment models being tested on a state-wide basis, and that the majority of states do 
not assess the educational outcomes of students with mild disabilities beyond what is 
required as reportable data by federal mandates. Those states that have shown 
exemplary efforts in outcomes assessments for students with mild disabilities have 
been supported by federal monies. For the most pan, although states are interested 
in outcomes assessment, few have the resources to do more than what is required by 
law. 

The outcomes assessment efforts of states were not found to correlate 
significantly with the performance chart variables of per pupil expenditure!, 
graduation rates, percentage of students with disabilities served, and percentage of 
minority student enrollment. Walther-Thomas concluded that states find "niches" of 
outcomes assessment efforts with which they arc comfortable, and collect data they 
find manageable. Most state agencies lack personnel with extensive evaluation and 
research skills to make wider use of the data that are collected. 

Finally, in addressing implications of the research, Walther-Thomas stresses 
that future efforts must consider teacher training, assessment procedures, 
organizational structure, and public awareness and involvement as focal points of 
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Table 7 

States with Outcomes Grants 



Funding Type of Sample/Target 

State Source Study Population Outcomes Variables 



NE State/Federal 
grant 



Outcomes/ 
PoBcy studies 



CA 



NH 



State grant 



Federal grant 



IA 



Federal grant 



Transition/ 
Follow-up 



Outcomes/ 
FeasWfity 
of statewide 
database 



Transition/ 
Follow-up 



KY State/Federal 
grant 



Ml 



State grant 



Former Students 
w/MUd, Moderate 
MR 



Former 
Special Ed 
students 

Former 
Special Ed 
students 



Former 
Special Ed 
students 

Former 
Special Ed 



Current and 
former 
Special Ed 
students 



Earnings 

Serf-report soda! adjustment 
Civic participation 
Involvement with law 
O ccu p a t ional status 
Status of independent fivlng 

Meaningful employment 
Quality of adult Bfe 



Attendance rates 
Retention 
Suspension 
Grade performance 
Withdrawal rates 
Student satisfaction 



Outcomes: 

Employment status 

Living arrangement 
Involvement In community 

social activities 
Vocational or social 

training programs 
Community-based 

instruction in school 
Family/community 

interaction 
Process: 

Type of special ed 

Familial involvement 

Vary according to 
disability and severity 
level 



change efforts. Change is difficult to initiate and hard to accept; states often consider 
themselves ill-prepared on many fronts to . effectively address all of these issues 
related to outcomes assessment of students with disabilities. Considerable time must 
be devoted to promoting the attitude that perceived barriers are surmountable in 
light of the many competing priorities of state agencies. 

Another recent doctoral dissertation investigating the variance of outcome 
assessment practices and data is that of Melissa Darrow (1990), who conducted a 
Delphi study about the theoretical models that are being used in school follow-up, 
follow-along research. In preparation for the Delphi, she reviewed all major follow- 
up follow-along studies of state-wide or national data. She then attempted to validate 
Hal pern's (1990) General Transition Follow- Along Model, and to arrive at some 
consensus on research questions to be common across state-wide studies. The Delphi 
survey was conducted as both a pilot and formal investigation. Darrow (1990) 
concluded that many professionals are reporting a great amount of data from follow- 
up, follow-along research, but that the lack of consistency and comparability across 
studies "prevents meaningful conclusions to be drawn concerning outcomes for 
students with disabilities on a nation-wide basis" (p. 69). What is agreed upon is that 
a universally accepted model to guide studies is needed. 

The General Transition Follow-Along Model (Halpern, 1990) is presented as a 
viable starting place for consensus-building on an outcomes model because it is 
longitudinal and balanced across important dimensions of community adjustment. 
Through the Delphi approach, the model's domains were supported as important by 
those who participated in the consensus-building process. Darrow (1990) also 
presented recommendations regarding possible revisions to the model, in an effort to 
advance the effective longitudinal analysis of special education programs. 

The growing awareness of the need for outcomes-based assessment has 
similarly been apparent in federal funding priorities. In 1988, the Office of Special 
Education and Rehabilitation Services established a funding priority for projects 
addressing transition follow-up and follow-along services. A total of 14 states 
(California, Colorado, Connecticut, Delaware, Florida, Hawaii, Kansas, Kentucky, 
Minnesota, New Hampshire, Oregon, Vermont, Washington and Wyoming), 
received three to four year grants designed to improve tracking systems and 
program options for students exiting special education. Each grant recipient was 
expected to develop, implement, and evaluate strategies to improve services for 
special education students that would enhance their participation in community life. 

Many state special education departments are taking active steps to begin to 
address the issue of how to mesh with the general education reform movement's 
increased requirements of students, and how to address the many issues that arise for 
students outside the mainstream of America's educational system and who comprise 
an extremely heterogeneous group of youngsters. The states are taking very 
different approaches to the major policy changes that are encompassed in the 
reform movement and to the push for national indicators of educational outcomes. A 
few states have been extremely active in developing policies related to outcomes 
assessment and indicators of outcomes for their students with disabilities. Among 
these states are California, Colorado, Connecticut, Hawaii, Iowa, Kentucky, Maryland, 
Michigan, Nebraska, New- Hampshire, and Oregon. 

California . Through a $500,000 state grant, California's Department of 
Education undertook a transition follow-along study designed to evaluate the long- 
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term effectiveness of special education programming. Initiated in 1986, the 
California Department of Education sought a systems change aimed at improving 
transition planning and employment outcomes by developing a formal model for 
transition and outcomes. The focus of California's work was on transition, every 
student having a well-planned Individual Transition Plan that included provisions 
for successful outcomes. Through a series of year-long meetings and public forums, 
California identified two broad domains for outcomes assessment: meaningful 
employment and quality of adult life, both defined from the vantage point of persons 
without disabilities. To assess these domains. California collected data on six areas 
identified by Oregon (i.e., student and family characteristics, school services 
received, school achievement, quality of life while in school, post-school services 
received, and quality of life out of school). 

California's conceptual model centered around the two outcomes and the 
planning necessary to achieve them. Separate responsibilities were detailed at the 
individual student, local education, and state education levels. A main concern 
throughout the project was to promote community involvement in follow-along 
services, thereby reducing the cost and enhancing the long-term effectiveness. To 
foster community participation, the Department of Education targeted all student 
outcomes, not just the outcomes of special education students. Integral to this process 
was the offering of multiple small grants for schools and communities to develop 
workable, community-based programs that improved long-term employment 
outcomes. Through these grants, California's aim was to promote long-term services 
that, enhance student outcomes in meaningful employment and quality of adult life. 

Colorado. Colorado's Department of Education (1988) received federal money 
for a project entitled "An Evaluation of the Effectiveness of Special Education 
Programming at the Secondary Level Based Upon Student Outcome and Program 
Quality Indicators." The project sought to accomplish four objectives: (1) to pilot and 
refine quality indicator instruments. (2) to study the relationship between student 
outcomes and the degree to which the quality indicators, conditions, practices, 
attitudes, and other factors are present in schools, (3) to provide reports for the 
participating schools on the assessed quality and outcomes for their own use in 
program improvement, and (4) to develop a model for state agencies and districts to 
follow in assessing program effectiveness. 

Colorado used the Mid-South Regional Resource Center (1986) document, 
Effectiveness Indicators for Special Educators, as the basis for its working 
assumptions and research questions. Under the direction of Dr. Mary Ann Lachat, of 
the Center for Resource Management (CRM) in New Hampshire. Colorado sought to 
"use and/or adapt an appropriate instrumentation and methodology that is being 
used by New Hampshire" in order to "maximize the use of resources and establish and 
implement consistent and compatible evaluation approaches" (p. 3). 

Colorado proposed to collect indicator data on both process and outcome 
indicators. Process indicator variables included attendance, suspension, dropout rate, 
participation in regular and vocational education programs, and integration in 
school and community settings. Outcome indicator variables included graduation 
rate, grades across curriculum areas, achievement of IEP objectives, satisfaction, 
self-concept, independent living skills, job preparation/employabiliiy knowledge, 
proportion of graduates with disabilities entering post-secondary education within 
two years of graduation, proportion of graduates who entered post-secondary 
education in the past two years who completed training or who are still in school, 



45 

GG 



survey of graduates with disabilities showing employment rate and income, and 
proportion of graduates who have been involved in the community by voting or by 
participating in organizations. 

Colorado has taken great care in specifying data collection procedures, 
sources, and instruments as well as appropriate analytical strategies. It has 
developed a well-planned technical framework and provides an example of a state- 
level emphasis on student outcome-based assessment that is thoughtfully organized 
and implemented. The state is primarily concerned with model development for 
future use at the district and state levels. 

Connecticut. Connecticut is one of several states that received federal monies 
to develop a statewide evaluation plan and indicator system of special education 
outcomes (Connecticut Department of Education, 19S8). Making the assumption that 
outcome measures for special education should be essentially the same as for regular 
education programs, Connecticut developed a Common Core of Learning (CCL) that 
identifies outcomes for all children. The CCL has three overall groups of outcomes 
(attitudes and attributes, skills and competencies, and understanding and application 
of competencies) organized into four categories for the special education student 
population (student participation, academic competencies, attitudes and attributes, 
and graduate follow-up data independent living skills). 

Connecticut used the CCL and Effectiveness Indicators for Special Education 
(Mid-South Regional Resource Center, 1986) as the basis for selecting outcomes. 
Currently, Connecticut is instituting three data collection strategies: longitudinal 
(academic competence, attitudes), periodic (participation, graduate/dropout rate), 
and one-time studies (graduate follow-up, community attitude). The state has 
designated which data types are already being collected (e.g., Connecticut Mastery 
Test), and described the desired data collection for outcome measures that are either 
inappropriate or not gathered at this time. For example, Connecticut is now 
promoting what it calls "out-of-level* testing, in which the level of mastery testing 
in which special education students participate is determined on an individual basis 
at the IEP meeting. Also, since no special education graduate follow-up data are 
currently collected on a statewide basis, plans are being made to develop instruments 
to do so, while considering pertinent differences between people with mild 
disabilities and people with substantial disabilities. Connecticut has been funded 
through OSERS since 1988 to develop mastery test data bases. 

Hawaii. Hawaii's tracking study is concentrating on the development of a 
theoretical model for transition planning. Currently in the development process, 
Hawaii plans to use the theoretical model as a framework from which to collect and 
analyze longitudinal transition data. The research team plans to develop a 
methodology that will circumvent the more common problems associated with follow- 
up studies: poor reliability, inadequate sampling, nebulous criteria, and biases in 
samples. Hawaii has targeted the following outcomes variables: current employment 
adjustment, post-secondary training, current living arrangements, 
social/recreational activities, financial status, type of secondary school program, and 
general problems encountered. 

Iowa . For five years, Iowa randomly selected half of ils special education 
students (1,000 per year) for its longitudinal research on transition planning. Using 
a personal interview format, the research team contacted participants three times: at 
one year, three years, and five years post-schooling. Iowa was concerned with 




students' outcomes in four areas: employment, living situation, ability to pay living 
expenses, and involvement in leisure activities. A successful graduate (i.e., one with 
successful outcomes) was originally defined as a former student who was employed 
full or part time: buying a borne* living independently or with a friend; paying more 
than half of his/her living expenses; and involved in more than three leisure 
activities. Iowa later extended this definition to include a broader range of individual 
circumstances. Nevertheless, Iowa maintained the notion of a successful educational 
outcomes as based on involvement in these four life activities. 

Iowa collected background information data from student records, on the type 
of high school programs students participated in, on students' own evaluations of 
their school experiences, on their current life circumstances (marital status, living 
arrangement, leisure activities), and on information about their past and current 
employment (e.g., type and location of job, salary, hours worked, length of 
employment, fringe benefits). Iowa is analyzing its data by disability and has already 
published separate reports for students with mental disabilities, behavior disorders, 
learning disabilities, and mild disabilities. Results of the research will be used to 
recommend changes in the state special education curriculum. 

Iowa's effort to achieve random sampling is noteworthy (along with 
Nebraska's) because it avoids a common limitation in many of the state outcome 
studies, that of self-selected samples of schools willing to cooperate with empirical 
research. This limitation is problematic in at least two ways. First, schools 
experiencing the greatest problems with students dropping out or performing poorly 
would be the least likely to cooperate and be evaluated by outside researchers. Thus, 
the data would tend to underrepresent the real issues at stake. Second, and related to 
this, recommendations and policies drawn from these data are likely to be ineffective 
for the range of students present in such schools. An effective model of outcomes 
assessment needs to address the experience of all students. 

Kentucky. Kentucky conducted a study from October, 1988 to March, 1990 to 
examine the status of former special education students and the relationship between 
special education programming and post-education outcomes. Under the direction of 
the Department of Education and the University of Kentucky, Kentucky's research 
used a sample of 1,250 former students served during the 1982-83 school year, and 
telephone interviews were used to collect the data. Kentucky's study examined five 
areas in relation to post-secondary school outcomes: (1) differing categorical 
placements, services delivered, and designs of service delivery system. (2) degree of 
participation in vocational education, (3) extent of transition planning, (4) degree of 
interaction between families and community agencies before exiting school, and (5) 
degree of participation in community instruction programs. 

Kentucky operationalized these concepts into both process and outcomes 
variables. Type of special education programming received and familial 
involvement in transition planning were among Kentucky's process variables. 
Outcomes variables included current employment status, living arrangement, 
involvement in community and social activities, the availability of vocational, 
training programs, the degree of community-based instruction in school, and the 
degree of interaction between families and community agencies during the 
transition period. Results from the Kentucky's research will be used to recommend 
changes in special education programming and in improving transition planning. 
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Maryland Maryland's tracking study, jointly directed by Maryland's 
Department of Education and the University of Maryland, focused on improving 
secondary special education programs to promote positive post-school outcomes and 
developing a system of tracking students exiting secondar y special education 
programs. Maryland's sample was comprised of students leaving special education 
programs either by graduating, aging out, or dropping out during the 1987-98 school 
year. A control group consisted of 480 regular education students not attending 
college. The study examined outcomes with regard to employment status, perceived 
co an ccUoa between training and work, job finding strategies, living situation, 
current employment stains, and level of satisfaction with services received. Data 
were collected through record reviews and telephone interviews with former 
students ana families. 

Michigan. Michigan has taken a conceptually different approach for 
developing a system of outcomes for students in special education. In contrast to 
identifying certain pertinent variables from which to evaluate outcomes, Michigan's 
approach is disability specific. Through an extensive process gathering 
rec o m mend ations from professionals and parents, Michigan concluded that a si- jle 
system of outcome indicators could not be created to measure students in both special 
and general education, nor even to cover the range of students in special education. 
To be comprehensive, a system of special education outcomes had to be specific to 
different disabilities and reflective of differing levels of severity. Identifying 12 
disability groups, Michigan expects to create two documents for each disability 
grsup: a document Identifying the unique educational needs of students with that 
Hxib) fry and a special education outcomes guide for that disability. The guides are 
<VnZ; j &ii to provide direction and focus to schools, not to prescribe minimum 
«/-.Mv*avy standards or to mandate best practices. 

tohigac has developed a five-step process for deriving appropriate outcomes 
for each disability ■ mp. At the first stage, a wide range of participants are selected 
for Input on needs M outcomes for a given disability group. The next stage involves 
the selection of appropriate experts to author a unique needs paper for the disability 
group. A Consensus Group is then identified and convened to generate basic outcomes 
for students in the disability group. The next step involves the identification of a 
Referent Group to be charged with refining the outcomes. Last, a Validation Group is 
convened for objective appraisal of the outcomes selected. 

Nebraska. Through a state and federal policy studies grant, Nebraska's 
Department of Education sought to identify factors influencing the educational 
outcomes for students with mild and moderate mental retardation and relate these to 
differences observed in educational programs. 

Using a relatively small sample, only 95 former students who were randomly 
selected to represent a cross-section of students in special education, Nebraska 
employed what it called a holistic approach to outcomes measurement. Paramount in 
the design of Nebraska's study was a link between post-school success and a 
satisfactory quality of life, or the extent to which a person is in control of her/his 
life. Nine post-school success variables (outcome variables) were employed, 
including independence in decision-making, independence in residence, social 
activities, personal satisfaction, integration on the job, independence on the job, 
performance ratings on the job, earned income, and degree of government support. 



Other areas examined were civic participation (voting), involvement with the 
law, and occupational status. Results indicated that two factors most positively 
correlated with the outcome measures: the students' level of functioning, and the 
degree of match between skills taught in school and those needed in life. Most 
notably, Nebraska found that post-school success was largely composed of job-related 
variables (i.e., earnings, independence of job tasks, and performance ratings on the 
job). Job success, thus, accounted for a significant portion of a former student's 
quality of life. 

New Hampshire. New Hampshire's Department of Education contracted with 
Dr. Mary Ann Lachat at the Center for Resource Management at the University of 
New Hampshire to direct its outcomes study. The study, negotiated to be a feasibility 
study, was designed to determine whether a state-wide data base could be designed 
and maintained that contained information on program effectiveness and student 
outcomes for special education. New Hampshire's sample was not randomly selected; 
rather, data were provided by schools volunteering to be part of the study. From this 
data, the research team concluded that state- wide data base could be maintained on 
the following outcomes: attendance rates, grade performance, retentions, suspension, 
withdrawal rates, and a number of student satisfaction measures (e.g.. evaluation of 
teacher). For other outcomes, however, the data were too erratic from school district 
to school district. 

Oregon. Oregon has been very systematic in its approach to outcomes. 
Contracting with Dr. Michael Benz and Dr. Andy Halpern at the University of Oregon. 
Oregon's Department of Education sought to develop and validate a comprehensive 
follow-along strategy and management information system for students leaving 
special education. Inherent in Oregon's model was a distinction between follow-up 
and follow-along studies. Follow-up studies were defined as cross-sectional, with data 
collected at one point in time. In contrast, follow-along referred to longitudinal 
studies assessing student progress at multiple points in time. Halpern (1990) 
concluded that the follow-along approach has greater utility because it tracks 
subjects across a longer period of time, provides more accurate information, and can 
provide reliable baseline data. 

Currently in the development phase, the research team began ihc process of 
establishing a conceptual model of follow-along from tbeir prior research and by 
identifying a series of research questions. A set of 68 questions was formulated to 
provide a framework for data collection and dissemination. These questions were 
grouped into six broad areas covering both variations in students while in school and 
the outcomes they achieve. The six general areas were student and family 
characteristics, school services received, school achievement, quality of life while in 
school, post-school services received, and quality of life out of school. Oregon plans 
to collect data in these areas using five instruments, including interviews of parent 
and student, interviews while in school and post-schooling, and a computer-assisted 
teacher questionnaire. The instruments are currently being developed. Data are 
presently being collected in Oregon arid Nevada. When complete, Oregon expects to be 
able to disseminate its model to a number of other states. 




Conclusion 



This paper has attempted to synthesize the literature and the activities of states 
in relation to assessing educational outcomes. It has been recognized throughout 
that we must know about what is happening in general educational policy and* state 
activities at the same time that we look at assessing educational outcomes for students 
with disabilities. This is a significant undertaking since the number of things going 
on and the speed with which they are occurring is tremendous. And, unfortunately, 
for the most part, this is all happening with minimal to no recognition of special 
education. If one looks at most of what is being said in the general education 
literature, the logical conclusion is that special education is a separate system that is 
taking care of itself. While general education policy makers are recognizing greater 
diversity in the student population, this does not include diversity to the extent 
covered by including students receiving special education services. 

When reading this paper, it is important to realize two things. ' First, its 
contents are out-of-date almost as soon as they are written. We have opted to update 
this synthesis annually. Clearly, it could be updated on a semi-annual or even 
quarterly basis. Second; this paper's contents can be characterized as "broad" rather 
than in-depth. The goal was to cover relevant topics in enough detail to convey their 
relevance to the issues surrounding the assessment of educational outcomes of 
students with disabilities. It has covered the many topics surrounding the broad 
notions of: 

• Educational Reform in the U.S. 

• Current Practice and Definitions 

• Current Models of Educational Indicators 

• Critical Issues to Consider in Developing a Comprehensive System of Outcome 
Indicators 

• Current Status of Outcome Indicator Activities in Policy Groups 

• Current Status of Outcome Indicator Activities in State Education Agencies 

Each of these area's many topics could be discussed in considerably more detail 
than is now done. Those topics that are most relevant to current activities will be 
pulled out and expanded into separate reports. Topics now being developed into 
reports are: 

• Exclusionary Policies in National Data Bases 

• Update on Stale Special Education Activities in Outcomes Assessment 

• Working Paper on a Conceptual Model of Educational Outcomes for Children 
and Youth with Disabilities 

In addition to updating a synthesis of the literature on outcomes and related 
issues, the National Center on Educational Outcomes is devoting its efforts to 
conducting annual surveys of state practice, developing consensus on an outcomes 
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model, identifying solutions to technical and implementation issues, and sharing 
information with many audiences involved in or interested in the assessment of 
educational outcomes. These integrated activities will help to ensure that students 
with disabilities are remembered as educational reform proceeds into the 21st 
century. 
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